Hardware-based techniques applicable for ray tracing for efficiently representing and processing an arbitrary bounding volume

ABSTRACT

A bounding volume is used to approximate the space an object occupies. If a more precise understanding beyond an approximation is required, the object itself is then inspected to determine what space it occupies. Often, a simple volume (such as an axis-aligned box) is used as bounding volume to approximate the space occupied by an object. But objects can be arbitrary, complicated shapes. So a simple volume often does not fit the object very well. That causes a lot of space that is not occupied by the object to be included in the approximation of the space being occupied by the object. Hardware-based techniques are disclosed herein, for example, for efficiently using multiple bounding volumes (such as axis-aligned bounding boxes) to represent, in effect, an arbitrarily shaped bounding volume to better fit the object, and for using such arbitrary bounding volumes to improve performance in applications such as ray tracing.

CROSS-REFERENCE TO RELATED PATENTS AND APPLICATIONS

This application is related to the following commonly-assigned USpatents and patent applications, the entire contents of each of whichare incorporated by reference:

-   -   U.S. application Ser. No. 14/563,872 titled “Short Stack        Traversal of Tree Data Structures” filed Dec. 8, 2014;    -   U.S. Pat. No. 9,582,607 titled “Block-Based Bounding Volume        Hierarchy”;    -   U.S. Pat. No. 9,552,664 titled “Relative Encoding For A        Block-Based Bounding Volume Hierarchy”;    -   U.S. Pat. No. 9,569,559 titled “Beam Tracing”;    -   U.S. Pat. No. 10,025,879 titled “Tree Data Structures Based on a        Plurality of Local Coordinate Systems”;    -   U.S. application Ser. No. 14/737,343 titled “Block-Based        Lossless Compression of Geometric Data” filed Jun. 11, 2015;    -   U.S. patent application Ser. No. 16/101,066 titled Method for        Continued Bounding Volume Hierarchy Traversal on Intersection        Without Shader Intervention;    -   U.S. patent application Ser. No. 16/101,109 titled “Method for        Efficient Grouping of Cache Requests for Datapath Scheduling”;    -   U.S. patent application Ser. No. 16/101,247 titled “A Robust,        Efficient Multiprocessor-Coprocessor Interface”;    -   U.S. patent application Ser. No. 16/101,180 titled        “Query-Specific Behavioral Modification of Tree Traversal”;    -   U.S. patent application Ser. No. 16/101,148 titled “Conservative        Watertight Ray Triangle Intersection”;    -   U.S. patent application Ser. No. 16/101,196 titled “Method for        Handling Out-of-Order Opaque and Alpha Ray/Primitive        Intersections”; and    -   U.S. patent application Ser. No. 16/101,232 titled “Method for        Forward Progress and Programmable Timeouts of Tree Traversal        Mechanisms in Hardware”.

FIELD

The present technology relates to computer graphics, and moreparticularly to ray tracers. More particularly, the technology relatesto hardware acceleration of computer graphics processing including butnot limited to ray tracing. The example non-limiting technology hereinalso relates to efficient ray-primitive wide complet based intersectiontests for geometry that does not tightly fit into single, axis-alignedbounding boxes.

BACKGROUND & SUMMARY

Real time computer graphics have advanced tremendously over the last 30years. With the development in the 1980's of powerful graphicsprocessing units (GPUs) providing 3D hardware graphics pipelines, itbecame possible to produce 3D graphical displays based on texture-mappedpolygon primitives in real time response to user input. Such real timegraphics processors were built upon a technology called scan conversionrasterization, which is a means of determining visibility from a singlepoint or perspective. Using this approach, three-dimensional objects aremodelled from surfaces constructed of geometric primitives, typicallypolygons such as triangles. The scan conversion process establishes andprojects primitive polygon vertices onto a view plane and fills in thepoints inside the edges of the primitives. See e.g., Foley, Van Dam,Hughes et al, Computer Graphics: Principles and Practice (2d Ed.Addison-Wesley 1995 & 3d Ed. Addison-Wesley 2014).

Hardware has long been used to determine how each polygon surface shouldbe shaded and texture-mapped and to rasterize the shaded, texture-mappedpolygon surfaces for display. Typical three-dimensional scenes are oftenconstructed from millions of polygons. Fast modern GPU hardware canefficiently process many millions of graphics primitives for eachdisplay frame (every 1/30^(th) or 1/60^(th) of a second) in real timeresponse to user input. The resulting graphical displays have been usedin a variety of real time graphical user interfaces including but notlimited to augmented reality, virtual reality, video games and medicalimaging. But traditionally, such interactive graphics hardware has notbeen able to accurately model and portray reflections and shadows.

There is another graphics technology which does perform physicallyrealistic visibility determinations for reflection and shadowing. It iscalled “ray tracing”. Ray tracing refers to casting a ray into a sceneand determining whether and where that ray intersects the scene'sgeometry. This basic ray tracing visibility test is the fundamentalprimitive underlying a variety of rendering algorithms and techniques incomputer graphics. Ray tracing was developed at the end of the 1960'sand was improved upon in the 1980's. See e.g., Appel, “Some Techniquesfor Shading Machine Renderings of Solids” (SJCC 1968) pp. 27-45;Whitted, “An Improved Illumination Model for Shaded Display” Pages343-349 Communications of the ACM Volume 23 Issue 6 (June 1980); andKajiya, “The Rendering Equation”, Computer Graphics (SIGGRAPH 1986Proceedings, Vol. 20, pp. 143-150). Since then, ray tracing has beenused in non-real time graphics applications such as design and filmmaking. Anyone who has seen “Finding Dory” (2016) or other Pixaranimated films has seen the result of the ray tracing approach tocomputer graphics—namely realistic shadows and reflections. See e.g.,Hery et al, “Towards Bidirectional Path Tracing at Pixar” (2016).

Generally, ray tracing is a rendering method in which rays are used todetermine the visibility of various elements in the scene. Ray tracingis a primitive used in a variety of rendering algorithms including forexample path tracing and Metropolis light transport. In an examplealgorithm, ray tracing simulates the physics of light by modeling lighttransport through the scene to compute all global effects (including forexample reflections from shiny surfaces) using ray optics. In such usesof ray tracing, an attempt may be made to trace each of many hundreds orthousands of light rays as they travel through the three-dimensionalscene from potentially multiple light sources to the viewpoint. Often,such rays are traced relative to the eye through the scene and testedagainst a database of all geometry in the scene. The rays can be tracedforward from lights to the eye, or backwards from the eye to the lights,or they can be traced to see if paths starting from the virtual cameraand starting at the eye have a clear line of sight. The testingdetermines either the nearest intersection (in order to determine whatis visible from the eye) or traces rays from the surface of an objecttoward a light source to determine if there is anything intervening thatwould block the transmission of light to that point in space. Becausethe rays are similar to the rays of light in reality, they makeavailable a number of realistic effects that are not possible using theraster based real time 3D graphics technology that has been implementedover the last thirty years. Because each illuminating ray from eachlight source within the scene is evaluated as it passes through eachobject in the scene, the resulting images can appear as if they werephotographed in reality. Accordingly, these ray tracing methods havelong been used in professional graphics applications such as design andfilm, where they have come to dominate over raster-based rendering.

Ray tracing can be used to determine if anything is visible along a ray(for example, testing for occluders between a shaded point on ageometric primitive and a point on a light source) and can also be usedto evaluate reflections (which may for example involve performing atraversal to determine the nearest visible surface along a line of sightso that software running on a streaming processor can evaluate amaterial shading function corresponding to what was hit—which in turncan launch one or more additional rays into the scene according to thematerial properties of the object that was intersected) to determine thelight returning along the ray back toward the eye. In classicalWhitted-style ray tracing, rays are shot from the viewpoint through thepixel grid into the scene, but other path traversals are possible.Typically, for each ray, the closest object is found. This intersectionpoint can then be determined to be illuminated or in shadow by shootinga ray from it to each light source in the scene and finding if anyobjects are in between. Opaque objects block the light, whereastransparent objects attenuate it. Other rays can be spawned from anintersection point. For example, if the intersecting surface is shiny orspecular, rays are generated in the reflection direction. The ray mayaccept the color of the first object intersected, which in turn has itsintersection point tested for shadows. This reflection process isrecursively repeated until a recursion limit is reached or the potentialcontribution of subsequent bounces falls below a threshold. Rays canalso be generated in the direction of refraction for transparent solidobjects, and again recursively evaluated. Ray tracing technology thusallows a graphics system to develop physically correct reflections andshadows that are not subject to the limitations and artifacts of scanconversion techniques.

Ray tracing has been used together with or as an alternative torasterization and z-buffering for sampling scene geometry. It can alsobe used as an alternative to (or in combination with) environmentmapping and shadow texturing for producing more realistic reflection,refraction and shadowing effects than can be achieved via texturingtechniques or other raster “hacks”. Ray tracing may also be used as thebasic primitive to accurately simulate light transport inphysically-based rendering algorithms such as path tracing, photonmapping, Metropolis light transport, and other light transportalgorithms.

The main challenge with ray tracing has generally been speed. Raytracing requires the graphics system to compute and analyze, for eachframe, each of many millions of light rays impinging on (and potentiallyreflected by) each surface making up the scene. In the past, thisenormous amount of computation complexity was impossible to perform inreal time.

One reason modern GPU 3D graphics pipelines are so fast at renderingshaded, texture-mapped surfaces is that they use coherence efficiently.In conventional scan conversion, everything is assumed to be viewedthrough a common window in a common image plane and projected down to asingle vantage point. Each triangle or other primitive is sent throughthe graphics pipeline and covers some number of pixels. All relatedcomputations can be shared for all pixels rendered from that triangle.Rectangular tiles of pixels corresponding to coherent lines of sightpassing through the window may thus correspond to groups of threadsrunning in lock-step in the same streaming processor. All the pixelsfalling between the edges of the triangle are assumed to be the samematerial running the same shader and fetching adjacent groups of texelsfrom the same textures. In ray tracing, in contrast, rays may start orend at a common point (a light source, or a virtual camera lens) but asthey propagate through the scene and interact with different materials,they quickly diverge. For example, each ray performs a search to findthe closest object. Some caching and sharing of results can beperformed, but because each ray potentially can hit different objects,the kind of coherence that GPU's have traditionally taken advantage ofin connection with texture mapped, shaded triangles is not present(e.g., a common vantage point, window and image plane are not there forray tracing). This makes ray tracing much more computationallychallenging than other graphics approaches—and therefore much moredifficult to perform on an interactive basis.

In 2010, NVIDIA took advantage of the high degree of parallelism ofNVIDIA GPUs and other highly parallel architectures to develop theOptiX™ ray tracing engine. See Parker et al., “OptiX: A General PurposeRay Tracing Engine” (ACM Transactions on Graphics, Vol. 29, No. 4,Article 66, July 2010). In addition to improvements in API's(application programming interfaces), one of the advances provided byOptiX™ was improving the acceleration data structures used for findingan intersection between a ray and the scene geometry. Such accelerationdata structures are usually spatial or object hierarchies used by theray tracing traversal algorithm to efficiently search for primitivesthat potentially intersect a given ray. OptiX™ provides a number ofdifferent acceleration structure types that the application can choosefrom. Each acceleration structure in the node graph can be a differenttype, allowing combinations of high-quality static structures withdynamically updated ones.

The OptiX™ programmable ray tracing pipeline provided significantadvances, but was still generally unable by itself to provide real timeinteractive response to user input on relatively inexpensive computingplatforms for complex 3D scenes. Since then, NVIDIA has been developinghardware acceleration capabilities for ray tracing. See e.g., U.S. Pat.Nos. 9,582,607; 9,569,559; US20160070820; US20160070767; and the otherUS patents and patent applications cited above.

A basic task for most ray tracers is to test a ray against allprimitives (commonly triangles in one embodiment) in the scene andreport either the closest hit (according to distance measured along theray) or simply the first (not necessarily closest) hit encountered,depending upon use case. The naïve algorithm would be an O(n)brute-force search. However, due to the large number of primitives in a3D scene of arbitrary complexity, it usually is not efficient orfeasible for a ray tracer to test every geometric primitive in the scenefor an intersection with a given ray.

By pre-processing the scene geometry and building a suitableacceleration data structure in advance, however, it is possible toreduce the average-case complexity to O(log n). Acceleration datastructures, such as a bounding volume hierarchy or BVH, allow for quickdetermination as to which bounding volumes can be ignored, whichbounding volumes may contain intersected geometric primitives, and whichintersected geometric primitives matter for visualization and which donot. Using simple volumes such as boxes to contain more complex objectsprovides computational and memory efficiencies that help enable raytracing to proceed in real time.

FIGS. 1A-1C illustrate ray tracing intersection testing in the contextof a bounding volume 208 including geometric mesh 320. FIG. 1A shows aray 302 in a virtual space including bounding volumes 310 and 315. Todetermine whether the ray 302 intersects geometry in the mesh 320, eachgeometric primitive (e.g., triangle) could be directly tested againstthe ray 302. But to accelerate the process (since the object couldcontain many thousands of geometric primitives), the ray 302 is firsttested against the bounding volumes 310 and 315. If the ray 302 does notintersect a bounding volume, then it does not intersect any geometryinside of the bounding volume and all geometry inside the boundingvolume can be ignored for purposes of that ray. Because in FIG. 1A theray 302 misses bounding volume 310, any geometry of mesh 320 within thatbounding volume need not be tested for intersection. While boundingvolume 315 is intersected by the ray 302, bounding volume 315 does notcontain any geometry and so no further testing is required.

On the other hand, if a ray such as ray 304 shown in FIG. 1B intersectsa bounding volume 310 that contains geometry, then the ray may or maynot intersect the geometry inside of the bounding volume so furthertests need to be performed on the geometry itself to find possibleintersections. Because the rays 304, 306 in FIGS. 1B and 1C intersect abounding volume 310 that contains geometry, further tests need to beperformed to determine whether any (and which) of the primitives insideof the bounding volume are intersected. In FIG. 1B, further testing ofthe intersections with the primitives would indicate that even thoughthe ray 304 passes through the bounding volume 310, it does notintersect any of the geometry the bounding volume encloses(alternatively, as mentioned above, bounding volume 310 could be furthervolumetrically subdivided so that a bounding volume intersection testcould be used to reveal that the ray does not intersect any geometry ormore specifically which geometric primitives the ray may intersect).

FIG. 1C shows a situation in which the ray intersects bounding volume310 and contains geometry that ray 306 intersects. To perform real timeray tracing, an intersection tester tests each geometric primitivewithin the intersected bounding volume 310 to determine whether the rayintersects that geometric primitive.

The acceleration data structure most commonly used by modern ray tracersis a bounding volume hierarchy (BVH) comprising nested axis-alignedbounding boxes (AABBs). The leaf nodes of the BVH contain the primitives(e.g., triangles) to be tested for intersection. The BVH is most oftenrepresented by a graph or tree structure data representation. In raytracing, the time for finding the closest (or for shadows, any)intersection for a ray is typically order O(log n) for n objects whensuch an acceleration data structure is used. For example, AABB boundingvolume hierarchies (BVHs) of the type commonly used for modern raytracing acceleration data structures typically have an O(log n) searchbehavior.

The BVH acceleration data structure represents and/or references the 3Dmodel of an object or a scene in a manner that will help assist inquickly deciding which portion of the object a particular ray is likelyto intersect and quickly rejecting large portions of the scene the raywill not intersect. The BVH data structure represents a scene or objectwith a bounding volume and subdivides the bounding volume into smallerand smaller bounding volumes terminating in leaf nodes containinggeometric primitives. The bounding volumes are hierarchical, meaningthat the topmost level encloses the level below it, that level enclosesthe next level below it, and so on. In one embodiment, leaf nodes canpotentially overlap other leaf nodes in the bounding volume hierarchy.

NVIDIA's RTX platform includes a ray tracing technology that bringsreal-time, cinematic-quality rendering to content creators and gamedevelopers. See https://developer.nvidia.com/rtx/raytracing. In many ormost implementations including NVIDIA RT Cores, the bounding volumessuch as shown in FIG. 1A-1C use axis-aligned bounding boxes (“AABBs”),which can be compactly stored and easily tested for ray intersection. Ifa ray intersects against the bounding box of the geometry, then theunderlying geometry is then tested as well. If a ray does not intersectagainst the bounding box of the geometry though, then that underlyinggeometry does not need to be tested. As FIGS. 1A-1C show, a hierarchy ofAABB's is created to increase the culling effect of a single AABBbounding box test. This allows for efficient traversal and a quickreduction to the geometry of interest.

As explained above, a bounding volume is often used to approximate thespace an object occupies. If a more precise understanding beyond anapproximation is required, the object itself is then inspected todetermine what space it occupies. Often, a simple volume (such as anaxis-aligned box) is used as bounding volume to approximate the spaceoccupied by an object. But objects can have arbitrary, complicatedshapes. So a simple volume often does not fit the object very well.Hence, although AABB's are widely and conventionally used for raytracing, not all geometry can be tightly fit to a single, axis-alignedbox. That causes a lot of space that is not occupied by the object to beincluded in the approximation of the space being occupied by the object.When geometry does not fit well, there can be false hits that lead toextra work. The example non-limiting technology has the advantage ofproviding single hit behavior for better fitting bounding volumes thancurrent ray tracing acceleration hardware. Previous solutions hadpotentially more false positives, repeated geometry, repeated tests, andexcess traversal costs. A multi-box solution supported by a widecompressed treelet format representation helps eliminate thatinefficiency.

Eventually the underlying geometry is not intersected, and no hit isreturned, but having to do those tests is inefficient compared to beingable to cull the work at a higher processing level. There are at leastthree problematic cases:

1. Triangles, likely long, skinny, and unaligned, that result in deadspace within the bounding volume.

2. Non-triangle geometry, like hair and curves, which do not fit well.

3. Geometry hidden under an instance transform.

In the triangle case, state of the art builders can split misbehavingtriangles across multiple boxes. Typically, the triangle itself is notbroken up, but instead is duplicated in the triangle buffer and includedunder multiple boxes. In this case, the intersection for a triangle canpotentially occur outside of its bounding box and can potentially betested against multiple times, but in all cases only one intersection isreturned. See e.g., Karras, Fast Parallel Construction of High-QualityBounding Volume Hierarchies, NVIDIA (2013); USP 10331632; Ganestem etal, “SAH guided spatial split partitioning for fast BVH construction”,Eurographics Volume 35 (2016), Number 2 (2016); Havran V., et al, “OnImproving KD-Trees for Ray Shooting”, In Winter School on ComputerGraphics pp. 209-217 (2002); Garanzha et al, “Grid-based SAH BVHconstruction on a GPU, The Visual Computer, Vol. 27, Issue 6-8, pp.697-706 (June 2011). In particular, triangle splitting without using theexample non-limiting technology herein can result in the sametriangle/primitive being returned multiple times. This isn't useful inpractice and can hurt performance, but it isn't necessarily harmful(aside from the performance impact) so long as the traversal algorithmon the processor side can handle it. Note that there is a mode select inthe DirectX (DXR) specification that allows for queries to have multiplereturns of the same triangle/primitive. If that mode select instead saysonly one intersection is allowed, then triangle splitting must bedisabled. One would do that if they were using an algorithm for whichmultiple returns would be too complicated to handle. Reference info onthe DXR “D3D12_RAYTRACING_GEOMETRY_FLAG_NO_DUPLICATE_ANYHIT_INVOC ATION”that can be found at

https://microsoft.github.io/DirectX-Specs/d3d/Raytracing.html#d3d12_raytracing_geometry_flags:

-   -   “By default, the system is free to trigger on any hit shader        more than once for a given ray-primitive intersection. This        flexibility helps improve the traversal efficiency of        acceleration structures in certain cases. For instance, if the        acceleration structure is implemented internally with bounding        volumes, the implementation may find it beneficial to store        relatively long triangles in multiple bounding boxes rather than        a larger single box.    -   However, some application use cases require that intersections        be reported to the any hit shader at most once. This flag        enables that guarantee for the given geometry, potentially with        some performance impact.    -   This flag applies to all geometry types.”        Example implementations herein allow a builder to use triangle        splitting while ignoring that flag.

In the non-triangle geometry case, procedural or complex geometry likehair and curves can use a separate intersection test(s) not acceleratedby the ray tracing acceleration hardware. This type of test typicallyrequires interruption of the traversal to return to a controlling orother specialized process (e.g., software running on a streaming orother multiprocessor) for a complex and expensive (e.g., software based)geometry-ray intersection test. The performance impact of interruptinghardware based traversal and in doing the complex test is high. Toreduce the impact, hair is often broken into smaller segments that aremore linear to better fit within a single box. Still, alignment issuescan cause this to be inefficient. For example, in the past, such smallersegments were treated as separate objects so that a ray along the curvecould positively test for bounding volume intersections multiple times,requiring multiple expensive ray-geometry intersection tests.

A third case is when instance transforms are used. A typicalacceleration structure comprises both a Top-Level Acceleration Structure(“TLAS”) and multiple Bottom-Level Acceleration Structures (“BLAS”). Intypical known implementations, the TLAS is in world space, while theBLASes use object space. The alignment then for a single object can bemore tightly fit in its own object space. The object is then placed in acommon world space via a transform. The same BLAS can be used multipletimes underneath a TLAS to represent multiple instances of thatgeometry. For example, a set of chairs in a room where all chairs areidentical can be represented by a single BLAS. The grouping of geometryin this way can be efficient for construction but can lead toinefficiency in traversal since the BLAS typically must be representedby a single AABB. For example, a four-legged chair bound by a single boxwould have dead space between legs as well as above the seat in front ofthe chair back. That dead space can lead to false hits of the instancein the TLAS—leading to extra transforms done into that object space onlyto miss at the BLAS level. Improvements can in some instances berealized by rotating the single AABB to provide a better fit, but thisdoes not work well for all geometry.

Given the great potential of a truly interactive real time ray tracinggraphics processing system for rendering high quality images ofarbitrary complexity in response for example to user input, further workis possible and desirable. In particular, it would be highly desirableto increase ray tracing efficiency and improve performance for geometricprimitives of arbitrary complexity.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A-1C show example simplified ray tracing tests to determinewhether the ray passes through a bounding volume containing geometry andwhether the ray intersects geometry within the bounding volume.

FIGS. 2A-2C, 2B1 and 2C1 illustrate example geometry that can be moretightly fit using multiple bounding volumes.

FIGS. 3A-3E illustrates how a single object can be encapsulated bymultiple bounding volumes of varying count.

FIGS. 4A-4C illustrates a simple stick figure enclosed by multiplebounding volumes in a bottom-level acceleration structure that can behoisted into a top-level acceleration structure for better cullingbefore transform.

FIGS. 5A, 5B illustrates an internal node bundling multiple curves intoone multi-box with each curve broken out separately in a child completat a lower layer.

FIG. 6 is a flowchart of an example non-limiting acceleration structurebuild process.

FIGS. 7A and 7B show example bounding volume hierarchy representations.

FIG. 8 shows example wide complet layout formats.

FIG. 9 illustrates an example non-limiting ray tracing graphics system.

FIG. 10 is a flowchart of an example non-limiting ray tracing graphicspipeline.

FIG. 11 is a flowchart of example non-limiting hardware based raytracing operations.

FIG. 12 shows a simplified example non-limiting traversal co-processorcomprising a tree traversal unit (TTU).

FIGS. 13 and 14 are flowcharts of example TTU processing.

FIG. 15 shows example wide complet intersection test hardware.

FIG. 16 shows an example comparator network layout.

FIG. 17 shows example pseudo-code for functions at the beginning of toplevel traversal.

FIGS. 18A and 18B illustrate more detailed ray tracing pipelines.

FIG. 19 is a flowchart of an example process to generate an image.

DETAILED DESCRIPTION OF NON-LIMITING EMBODIMENTS

The technology herein thus provides hardware-based capabilities thatefficiently accelerate ray tracing for such non-fitting geometry.Hardware-based technique are disclosed herein, for example, forefficiently using multiple bounding volumes (such as axis-alignedbounding boxes) to represent, in effect, an arbitrarily shaped boundingvolume to better fit the object, and for using such arbitrary boundingvolumes to improve performance in applications such as ray tracing. Inone embodiment, multiple bounding volumes constructed around an objectare treated as a single bounding volume for purposes of ray-volumeintersection testing. Multiple positive ray-volume intersection resultsbased on testing such multiple bounding volumes are distilled down intoa single ray intersection indication (another way of looking at it: ifthe ray hits any of the multiple bounding volumes, it hits all of themsince the multiple bounding volumes are known to contain the sameextended geometric primitive). This can result in substantialperformance increases for certain objects and scenes that previouslyrequired expensive multiple ray-object intersection tests.

In one example non-limiting embodiment, an AABB acceleration structureis constructed using compressed treelets (“complets”) that are wide,allowing multiple (e.g., up to 12 in some embodiments) bounding volumechildren to be tested simultaneously. The technology herein introduces amulti-box complet format that allows a single child to have multiplebounding volumes (e.g., AABBs) associated with it where a “hit”(positive result of a ray-bounding volume intersection test) on anypositive number (one or more) of those bounding volumes is treated as asingle “hit” or intersection for that child. In other words, thetraditional ray-bounding volume intersection test is in appropriatecases (i.e., cases where bounding the geometry doesn't fit the AABBbounding volumes well) explicated to test (e.g., in parallel within asingle cycle in some embodiments) a plurality of bounding volumes (allof which can be AABBs) instead of a single bounding volume. Thoseplurality of AABB bounding volumes are positioned, sized and oriented tomore closely enclose such geometry than any single AABB could. Theplurality of bounding volumes can have the same sizes and/or shapesand/or orientation or different sizes and/or shapes and/or orientations.They can be connected, overlapping or disjoint. This allowsapproximating non-box like shapes or otherwise non-axis-oriented shapeswith AABBs that result in a better fitting bounding volume than a singleAABB would.

Furthermore, in some embodiments, the geometry intersection testing inthe event of a hit between the ray and one of the explicated pluralityof bounding volumes can proceed in the same way as if a single AABB hadbeen used and found to intersect the ray. Increased efficiency resultsfrom a potential decrease in the number of hits between a set of raysand the explicated plurality of bounding volumes (AABBs) as compared tonumber of hits between that set of rays and a single, larger AABB.Performance does not suffer, especially in example embodiments thatessentially or actually test all of the explicated plurality of boundingvolumes in parallel so the amount of time needed to perform ray-boundingvolume intersection tests for all of the explicated plurality ofbounding volumes is on the same order as the amount of time to test asingle, larger AABB. Furthermore, in some example embodiments, an “earlyout” algorithm could eliminate the need to test further multi-boxes foran object once an intersection is found with one of the multi-boxes.

By way of further illustration, FIGS. 2A, 2B and 2C show examples ofgeometry that does not fit well into a single AABB. FIG. 2A shows acylinder, FIG. 2B shows a curved object, and FIG. 2C shows a triangle.These Figures show two-dimensional views, but the actual geometry andbounding volumes in modern ray tracing systems will bethree-dimensional. FIGS. 2B1 and 2C1 show 3D views of the FIGS. 2B and2C scenarios, respectively.

In these examples:

The cylinder of FIG. 2A would fit closely within a single AABB if thecylinder were vertically or horizontally aligned, but is a bad fit forany AABB because the cylinder's orientation is at an angle with respectto the axes.

The curved object of FIGS. 2B/2B1 is shaped such that different portionshave different orientations and no single AABB that encloses the objectwill provide a close fit.

The triangle shown in FIG. 2C/2C1 is not aligned with one of the axes,and its shape and dimensions are such that any AABB will also encloselots of space the triangle does not occupy.

In each case, an original AABB bounding box represented by the dottedlines can be used to encompass the geometry but would also encompasslarge amounts of empty space. As FIGS. 2A-2C, 2B1, 2C1 show, in eachcase the particular geometry can be more tightly fit with multiple(e.g., 4) smaller boxes represented by the solid-line boxes. In the FIG.2A configuration, all of the smaller boxes are the same size andorientation and are just placed at different positions. In the FIG.2B/2B1 configuration and the FIG. 2C/2C1 configuration, the smallerboxes are of different sizes and orientations.

In some cases such as FIGS. 2A and 2C, the smaller boxes overlap oneanother; in other cases, the do not overlap. There is no substantialpenalty to such overlapping for purposes of a ray intersection testsince in example embodiments, any number of intersections withindividual smaller bounding volumes will be resolved to a singleray-bounding volume intersection “hit” that will lead to testing the rayagainst the enclosed geometry. But FIG. 2B shows that the smallerbounding volumes need not overlap, and in some cases (e.g., if thegeometry has a void or gap, is defined as particles of a particularsystem e.g., to define smoke, fog, etc.), it may be desirable for thebounding volumes to be disjoint.

In example embodiments, the number of bounding boxes in a set isprogrammatic and controlled per complet. It could consist of any number,but for ease of implementation one example non-limiting embodimentchooses the even divisions of the complet width: 2, 3, 4, 6, and 12boxes per child. In the case of 2 boxes per child, each complet that canrepresent 12 boxes would have 6 children. For 3 boxes per child, eachcomplet that can represent 12 boxes would have 4 children. And so on, upto 12 boxes per child representing just a single child in that complet.Different numbers of child bounding volumes can be used depending on theparticular geometry and the particular implementation.

FIGS. 3A-3E show how the same object (in this case the cylinder of FIG.2A) can be encapsulated by multiple boxes of varying count, where againthe solid boxes represent the constituent boxes of the multi-box set ofan explicated plurality of bounding volumes while the dotted boxrepresents what would have been the singular bounding box. FIG. 3C isessentially the same scenario as FIG. 2A. However, FIGS. 3A and 3B showuse of fewer (e.g., 2 or 3) AABBs to bound the cylinder, and FIGS. 3Dand 3E show use of more (e.g., 6 or 12) AABBs. It can be seen that forlong thin non-aligned objects, closer fits (and thus smaller numbers offalse ray intersection hits) can be attained by using more boundingvolumes.

For long, skinny, unaligned triangles, the multi-box solution looks verymuch like conventional triangle splitting in that the same triangle isincluded under (within) multiple boxes. But unlike most conventionaltriangle splitting approaches, the multi-box scenario shown in FIGS.3A-3E as implemented by example non-limiting embodiments herein does notrequire duplicating the primitive. Specifically, it is conventional touse multiple bounding volumes instead of one large bounding volume tomore closely fit such triangles, but in such cases designers do notsplit the triangle into multiple parts because this can result in holesin the geometry and introduce artifacts in the resulting images.Accordingly, prior approaches duplicated the triangle for each smallerbounding volume, which could result in multiple ray-geometryintersection tests on each copy of the triangle in the case of rays forexample that are aligned with the long axis of the triangle.Furthermore, duplicating the triangle as in prior approaches requiredthe same triangle to be stored multiple times, resulting in increasedmemory usage. In example non-limiting approaches of multi-box, incontrast, the triangle will (need) not be stored more than once ortested more than once against the ray in a ray-triangle intersectiontest, realizing increased efficiency.

For hair and curves such as in FIG. 2B/2B1, the multi-box solutionallows for a better fitting volume reducing the false positives that canresult in a traversal interruption and costly return to the streamingmultiprocessor or other controlling software process. A trianglesplitting like solution could also be applied to curves, but it resultsin multiple returns of the same underlying object to the controllingprocess. The multi-box solution herein allows for just a single returnfor encapsulated shapes of arbitrary complexity.

Instance Transforms

In many modern ray tracing systems, objects replicated several or manytimes in the scene at different positions, orientations and scales canbe represented in the scene as instance nodes. These instance nodesassociate a bounding box and leaf node in the world space BVH with atransformation that can be applied to the world-space ray to transformit into an object coordinate space, and a pointer to an object-spaceBVH. Instance transforms avoid replicating the object space BVH datamultiple times in world space, saving memory and associated memoryaccesses. The instance transform increases efficiency by transformingthe ray into object space instead of requiring the geometry or thebounding volume hierarchy to be transformed into world (ray) space andis also compatible with additional, conventional rasterization processesthat graphics processing performs to visualize the primitives.

In such instance transform cases, multi-box can also be used forinstance “hoisting” from a lower level of the acceleration structure toa higher level of the acceleration structure. For example, the boundingboxes in a BLAS may be “hoisted” into the TLAS for better culling beforethe transform.

FIG. 4A shows a simple stick figure covered by 6 boxes in a BLAS inobject or instance space. Without multi-box in FIG. 4B, the TLAS (inworld space) would treat the 6 boxes as a single box (dashes). Withexample non-limiting implementations of multi-box, we can essentiallypull or hoist those 6 children defined in object space up into the TLASof the acceleration data structure as a 6-box multi-box child (solidlines in FIG. 4C) in world space—which should have fewer false hitsrequiring the traversal process to descend into the instance, therebyavoiding unnecessary transforms just to test rays against what amountsto dead space. Because the multi-box intersection testing can now beperformed using multi-box on the TLAS level in world space, thetransform of the ray to object space by the TTU can be avoided forpurposes of the multi-box/ray intersection test. Furthermore, themulti-boxes offer the additional advantage that they can more tightlyfit the geometry in world space than a conventional (e.g., dotted line)single AABB bounding box can. There isn't a rotation here, but one couldimagine the figure as rotated so those top-level boxes in FIG. 4C don'thave the same orientations as the root in the BLAS shown in FIG. 4A. Ifthe multi-boxes on the TLAS level are sufficiently reliable, then thereis no need to use the same multi-boxes at the root of the BLASlevel—although the multi-box technique can be used on the BLAS level forother intersection testing (e.g., to test for intersection with curvesdefining hair on the head of the stick figure shown). This is what FIGS.4A-4C show.

Furthermore, it is possible for the multi-boxes on the TLAS level ofFIG. 5B to be defined in ways other than by a builder transforming theFIG. 5A multi-boxes from object space to world space. For example, toconstruct the FIG. 5B multi-boxes, the builder could transform theunderlying geometry to world space and then construct TLAS world spacemulti-boxes to more tightly fit the transformed geometry.

Internal Nodes

In example embodiments, the multi-box complet format is not reservedjust for leaf nodes but can also be used for internal nodes. That iseffectively agglomerating multiple primitives or boxes into one bundle.FIG. 5A shows an internal node bundling multiple curves into onemulti-box child. If an intersection test against the multi-box structureshown in FIG. 5A reveals an intersection between a ray and any of themulti-boxes, a downward traversal of the BVH to a leaf node can provideadditional multi-boxes—and in this case a different set of multi-boxesfor each different primitive bounded by the FIG. 5A multi-boxes. Thus,FIG. 5B shows each curve broken out separately in a child complet at alower layer of the BVH, with a different set of AABB bounding boxesbeing used to bound each different curve or other primitive for purposesof ray-complet intersection testing. In this example, note that thebounding boxes of FIG. 5B are not further subdivisions of the FIG. 5Abounding boxes. The FIG. 5B boxes are often fully contained within theFIG. 5A bounding boxes while being further reduced in dimensions to moretightly fit associated primitives, but they do not need to be. Forexample, boxes on different layers can have different precisions so thatboxes on upper layers can be smaller than boxes on the lower layer.Because an upper layer ray intersection test will fail if the ray doesnot intersect a smaller upper layer box, no traversal performancedegradation results by making lower layer boxes larger than upper layerboxes. Rather, in some embodiments, the ray-complet intersection testperformed on a leaf node is a Boolean union with previously-performedintersection tests on upper layer bounding boxes for the same geometrybecause the traversal will not proceed downward to a leaf node if a testperformed on an upper layer does not show intersection.

The following provides more detail on how to construct BVH accelerationstructures that supports/provides multi-box and can be encoded usingwide complet formats. Following that, an overall hardware based realtime graphics processing system including real time ray tracing hardwarethat can traverse and manipulate such acceleration structures isdisclosed.

Building A Bounding Volume Hierarchy

As described above, an acceleration data structure comprises a hierarchyof bounding volumes (bounding volume hierarchy or BVH) that recursivelyencapsulates smaller and smaller bounding volume subdivisions. Thelargest volumetric bounding volume may be termed a “root node.” Thesmallest subdivisions of such hierarchy of bounding volumes (“leafnodes”) contain items. The items could be primitives (e.g., polygonssuch as triangles) that define surfaces of the object. Or, an item couldbe a sphere that contains a whole new level of the world that exists asan item because it has not been added to the BVH (think of the collarcharm on the cat from “Men in Black” which contained an entire miniaturegalaxy inside of it). If the item comprises primitives, the traversalco-processor upon reaching an intersecting leaf node tests rays againstthe primitives associated with the leaf node to determine which objectsurfaces the rays intersect and which object surfaces are visible alongthe ray.

Building a BVH can occur in two parts: static and dynamic. In manyapplications, a complex scene is preprocessed and the BVH is createdbased on static geometry of the scene. Then, using interactive graphicsgeneration including dynamically created and manipulated moving objects,another part of the BVH (or an additional, linked BVH(es) can be builtin real time (e.g., in each frame) by driver or other software runningon the real time interactive graphics system. BVH construction need notbe hardware accelerated (although it may be in some non-limitingembodiments) but may implemented using highly-optimized softwareroutines running on SMs 132 and/or CPU 120 and/or other developmentsystems e.g., during development of an application.

The first stage in BVH acceleration structure construction acquires thebounding boxes of the referenced geometry (FIG. 6, 204). This isachieved by executing for each geometric primitive in an object abounding box procedure that returns a conservative axis-aligned boundingbox (AABB) for its input primitive. Aligning bounding boxes with theaxes of the relevant coordinate systems for the geometry provides forincreased efficiency of real time geometrical operations such asintersection testing and coordinate transforms as compared for exampleto oriented bounding boxes (OBB's), bounding spheres, or otherapproaches. However, those skilled in the art will understand that theexample non-limiting multi-box approaches and associated wide completrepresentations described herein can also be applied to more expensivebounding constructs such as OBBs, bounding spheres and other boundingvolume technology. On the other hand, depending on the shape of theunderlying geometry, an aggregation of plural AABBs using the multi-boxapproach described herein may provide improvements over bothconventional AABB technology and other bounding technology such as OBBs.

The standard procedure will typically yield the dotted line boundingboxes of FIGS. 2A, 2B, 2C. Already subdivided bounding volumes that doinclude at least one portion of the geometry in a scene can be stillfurther recursively subdivided—like the emergence of each of asuccession of littler and littler cats from the hats of Dr. Seuss's’ TheCat In The Hat Comes Back (1958). The number and configurations ofrecursive subdivisions will depend on the complexity and configurationof the 3D object being modeled as well as other factors such as desiredresolution, distance of the object from the viewpoint, etc. One examplesubdivision scheme is a so-called 8-ary subdivision or “octree” in whicheach volume is subdivided into eight smaller volumes of uniform size,but many other spatial hierarchies and subdivision schemes are knownsuch as a binary tree, a four-ary tree, a k-d tree, a binary spacepartitioning (BSP) tree, and a bounding volume hierarchy (BVH) tree. Seee.g., U.S. Pat. No. 9,582,607.

At some level of subdivision (which can be different levels fordifferent parts of the BVH), the BVH construction process encountersgeometry making up the encapsulated object being modeled. Using theanalogy of a tree, the successive volumetric subdivisions are the trunk,branches, boughs and twigs, and the geometric is finally revealed at thevery tips of the tree, namely the leaves. At this point, the BVHconstruction process for example non-limiting embodiments hereinperforms an optimization at this stage to spot, using heuristic or otheranalytical techniques (which might include artificial intelligenceand/or neural networks in some embodiments), those leaf nodes that (likethe FIG. 2A/2B/2C scenarios) present poor fits with respect to thegeometry they contain (FIG. 6, 206). These poor-fitting nodes can beoptimized by applying multi-box as explained above, providing moreefficient AABBs—essentially collections of overlapping ornon-overlapping AABBs that create new bounding volume shapes that moretightly enclose geometry than a standard conservatively-constructed AABBcan (and without requiring the additional complexity needed to supportnon-axis aligned bounding volumes such as OBBs) (FIG. 6, 208).

This process continues until all bounding volumes containing geometryhave been sufficiently subdivided to provide a reasonable number ofgeometric primitives per bounding box (FIG. 6, 210). In examplenon-limiting embodiments, multi-boxes are not further subdivided (evenif applied to interior or non-leaf nodes). The real time ray tracer thatuses the BVH will determine ray-primitive intersections by comparing thespatial xyz coordinates of the vertices of each primitive with the xyzcoordinates of the ray to determine whether the ray and the surface theprimitive defines occupy the same space. The ray-primitive intersectiontest can be computationally intensive because there may be manytriangles to test. In many cases, it may be more efficient to furthervolumetrically subdivide and thereby limit the number of primitives inany “leaf node” to something like 16 or fewer.

The resulting compressed tree comprising compressed treelets is writtenout into a data structure in memory for later use by the graphicsprocessing hardware/software during e.g., real time graphics processingthat includes real time ray tracing (FIG. 6, 212).

FIGS. 7A and 7B show a recursively-subdivided bounding volume of a 3Dscene (FIG. 7A) and a corresponding tree data structure (FIG. 7B) thatmay be accessed by the ray tracer and used for hardware-acceleratedoperations. The tree data structure may be stored in memory andretrieved on demand based on queries.

The division of the bounding volumes may be represented in ahierarchical tree data structure with the large bounding volumerepresented by a parent node of the tree and the smaller boundingvolumes represented by children nodes of the tree that are contained bythe parent node. The smallest bounding volumes are represented as leafnodes in the tree and identify one or more geometric primitivescontained within these smallest bounding volumes.

The tree data structure includes a plurality of nodes arranged in ahierarchy. The root nodes N1 of the tree structure correspond tobounding volume N1 enclosing all of the primitives O1-O8. The root nodeN1 may identify the vertices of the bounding volume N1 and childrennodes of the root node.

In FIG. 7A, bounding volume N1 is subdivided into bounding volumes N2and N3. Children nodes N2 and N3 of the tree structure of FIG. 7Bcorrespond to and represent the bounding volumes N2 and N3 shown in FIG.7A. The children nodes N2 and N3 in the tree data structure identify thevertices of respective bounding volumes N2 and N3 in space. Each of thebounding volumes N2 and N3 is further subdivided in this particularexample. Bounding volume N2 is subdivided into contained boundingvolumes N4 and N5. Bounding volume N3 is subdivided into containedbounding volumes N6 and N7. Bounding volume N7 include two boundingvolumes N8 and N9. Bounding volume N8 includes the triangles O7 and O8,and bounding volume N9 includes leaf bounding volumes N10 and N11 as itschild bounding volumes. Leaf bounding volume N10 includes a primitiverange (e.g., triangle range) O10 and leaf bounding volume N11 includesan item range O9. Respective children nodes N4, N5, N6, N8, N10 and N11of the FIG. 7B tree structure correspond to and represent the FIG. 7Abounding volumes N4, N5, N6, N8, N10 and N11 in space.

The FIG. 7B tree in this particular example is only three to six levelsdeep so that volumes N4, N5, N6, N8, N10 and N11 constitute “leafnodes”—that is, nodes in the tree that have no child nodes. FIG. 7Ashows that leaf node bounding volumes N4, N6, and N8 each contains twotriangles of the geometry in the scene. For example, volumetricsubdivision N4 contains triangles O1 & O2; volumetric subdivision N6contains trials O5 & O6; and volumetric subdivision N8 containstriangles O7 & O8. FIG. 7A further shows that leaf node bounding volumeN5 contains a single cylinder O3 such as shown in FIG. 2A that does notprovide a good fit for the AABB bounding volume N5 shown in dottedlines. Accordingly, in an example non-limiting embodiment herein,instead of using the larger AABB bounding volume N5 for the ray-boundingvolume intersection test, TTU 138 instead tests the ray against aplurality of smaller AABB bounding volumes that are arranged,positioned, dimensioned and oriented to more closely fit cylinder O3.

The tree structure shown in FIG. 7B represents these leaf nodes N4, N5,N6, and N7 by associating them with the appropriate ones of primitiveO1-O8 of the scene geometry. To access this scene geometry, the TTU 138traverses the tree data structure of FIG. 7B down to the leaf nodes. Ingeneral, different parts of the tree can and will have different depthsand contain different numbers of primitives. Leaf nodes associated withvolumetric subdivisions that contain no geometry need not be explicitlyrepresented in the tree data structure (i.e., the tree is “trimmed”).

According to some embodiments, the subtree rooted at N7 may represent aset of bounding volumes or BVH that is defined in a different coordinatespace than the bounding volumes corresponding to nodes N1-N3. Whenbounding volume N7 is in a different coordinate space from its parentbounding volume N3, an instance node N7′ which provides the raytransformation necessary to traverse the subtree rooted at N7, mayconnect the rest of the tree to the subtree rooted at N7. Instance nodeN7′ connects the bounding volume or BVH corresponding to nodes N1-N3,with the bounding volumes or BVH corresponding to nodes N7 etc. bydefining the transformation from the coordinate space of N1-N3 (e.g.,world space) to the coordinate space of N7 etc. (e.g., object space). Asdescribed herein, example embodiments herein can “hoist” multi-boxesfrom the (object) coordinate space of N7 to the (world) coordinate spaceof N1-N3 in order to in some cases avoid the need for performing suchtransforms.

In example embodiments, the complet formatting performed by the processto write out BVH nodes is extended to include new, wider formats thatspecify the boxes per child. Sets are made of contiguous children. FIG.8 shows example layout formats for multi-box. In one embodiment, theselayout formats are sized the same way as legacy formats to fit within asingle cache line. In other implementations, the complet formats couldhave variable instead of fixed lengths to accommodate more entries forthose nodes that use multi-box. Furthermore, while compression is usedto reduce the size of the acceleration structure in example embodiments,other embodiments might not use compression or the same type ofcompression.

An example legacy format would provide one box each for child 0-child11. As FIG. 8 shows, for complets with 2 boxes per node, a 2-boxmulti-box format would use what was child 0 and child 1 before (in alegacy format) as child 0 and then its extension child 0 box 1Similarly, child 2 would be provided with an extension child 2 box 1,and so on. The complets meanwhile can contain the same additionalinformation used in legacy complet formats. Using this new multi-boxformat, the same data block that in legacy scenarios could represent 12child bounding volumes can now only represent 6 child boundingvolumes—half as many. Accordingly, in example embodiments, children 1,3, 5, 7, 9 and 11 are omitted from this format to make room for theextension child boxes. Those omitted children can be represented in asimilar multi-box format each with their own extension boxes, or theycan be represented without multibox if multibox is not needed for thegeometry they contain.

For complets with 3 boxes per node, the node corresponding to child 0 isrepresented as child 0, child 0 box 1, and child 0 box 2 Similarly, thenode corresponding to child 3 is represented as child 3, child 3 box 1,and child 3 box 2. Four-box complets represent each child with fourboxes, six-box complets represent each child with six boxes, andtwelve-box complets represent each child with twelve boxes. In theexample embodiment, a 12-box complet such as shown in FIG. 3E would takeup the entire data block space such that the format would be used onlyrepresenting the 12 multi-boxes of the same child node (0).

In example embodiments, all multi-boxes represented in one block can betested in parallel, although this is not a limitation. Other multiboxformats can be provided with some unused fields within the confines of asingle cache line sized block (i.e., embodiments can have numbers ofchild nodes that are not divisors of 12.) It is also possible to have amulti-box format where different children are defined to have differentnumbers of extension boxes. In terms of selecting which of the FIG. 8formats to use in a particular situation, parallel testing of each boxin the format means that processing performance will not be reduced byusing fewer rather than more boxes per child, but memory bandwidth canbe conserved if a smaller number of multiboxes are used for a givenprimitive consistent with desired accuracy (see FIGS. 3A-3E).

In one example non-limiting embodiment, a multi-box child represented bythe FIG. 8 formats is valid or invalid based solely on the primary childfields having an inverted zlo/zhi as in the legacy TTU behavior fornon-multi-box complets.

The FIG. 8 formats show header information including a field thatindicates the multi-box count and field that contain inheritance(ancestor/descendant) pointers into the BVH; pointers to each “child”volume including references to multiple boxes per child; and extendedheader information.

Example Real Time Ray Tracing Hardware Accelerated Implementation

The acceleration structure constructed as described above can be used toadvantage by software based graphics pipeline processes running on aconventional general purpose computer. However, the presently disclosednon-limiting embodiments advantageously implement the above-describedtechniques in the context of a hardware-based graphics processing unitincluding a high performance processors such as one or more streamingmultiprocessors (“SMs”) and one or more traversal co-processors or “treetraversal units” (“TTUs”)—subunits of one or a group of streamingmultiprocessor SMs of a 3D graphics processing pipeline, that is capableof providing wide complet intersection tests on multi-box boundingvolumes as described herein. The following describes the overallstructure and operation of such as system including a TTU 138 thataccelerates certain processes supporting interactive ray tracingincluding ray-bounding volume intersection tests, ray-primitiveintersection tests and ray “instance” transforms for real time raytracing and other applications. Such TTU has been enhanced to supportwide complets and multi-box ray-volume intersection testing as describeabove.

Example System Block Diagram

FIG. 9 illustrates an example real time ray interactive tracing graphicssystem 100 for generating images using three dimensional (3D) data of ascene or object(s) including the acceleration data structure constructedas described above.

System 100 includes an input device 110, a processor(s) 120, a graphicsprocessing unit(s) (GPU(s)) 130, memory 140, and a display(s) 150. Thesystem shown in FIG. 9 can take on any form factor including but notlimited to a personal computer, a smart phone or other smart device, avideo game system, a wearable virtual or augmented reality system, acloud-based computing system, a vehicle-mounted graphics system, asystem-on-a-chip (SoC), etc.

The processor 120 may be a multicore central processing unit (CPU)operable to execute an application in real time interactive response toinput device 110, the output of which includes images for display ondisplay 150. Display 150 may be any kind of display such as a stationarydisplay, a head mounted display such as display glasses or goggles,other types of wearable displays, a handheld display, a vehicle mounteddisplay, etc. For example, the processor 120 may execute an applicationbased on inputs received from the input device 110 (e.g., a joystick, aninertial sensor, an ambient light sensor, etc.) and instruct the GPU 130to generate images showing application progress for display on thedisplay 150.

Images generated applying one or more of the techniques disclosed hereinmay be displayed on a monitor or other display device. In someembodiments, the display device may be coupled directly to the system orprocessor generating or rendering the images. In other embodiments, thedisplay device may be coupled indirectly to the system or processor suchas via a network. Examples of such networks include the Internet, mobiletelecommunications networks, a WIFI network, as well as any other wiredand/or wireless networking system. When the display device is indirectlycoupled, the images generated by the system or processor may be streamedover the network to the display device. Such streaming allows, forexample, video games or other applications, which render images, to beexecuted on a server or in a data center and the rendered images to betransmitted and displayed on one or more user devices (such as acomputer, video game console, smailphone, other mobile device, etc.)that are physically separate from the server or data center. Hence, thetechniques disclosed herein can be applied to enhance the images thatare streamed and to enhance services that stream images such as NVIDIAGeForce Now (GFN), Google Stadia, and the like.

Furthermore, images generated applying one or more of the techniquesdisclosed herein may be used to train, test, or certify deep neuralnetworks (DNNs) used to recognize objects and environments in the realworld. Such images may include scenes of roadways, factories, buildings,urban settings, rural settings, humans, animals, and any other physicalobject or real-world setting. Such images may be used to train, test, orcertify DNNs that are employed in machines or robots to manipulate,handle, or modify physical objects in the real world. Furthermore, suchimages may be used to train, test, or certify DNNs that are employed inautonomous vehicles to navigate and move the vehicles through the realworld. Additionally, images generated applying one or more of thetechniques disclosed herein may be used to convey information to usersof such machines, robots, and vehicles.

Based on execution of the application on processor 120, the processormay issue instructions for the GPU 130 to generate images using 3D datastored in memory 140. The GPU 130 includes specialized hardware foraccelerating the generation of images in real time. For example, the GPU130 is able to process information for thousands or millions of graphicsprimitives (polygons) in real time due to the GPU's ability to performrepetitive and highly-parallel specialized computing tasks such aspolygon scan conversion much faster than conventional software-drivenCPUs. For example, unlike the processor 120, which may have multiplecores with lots of cache memory that can handle a few software threadsat a time, the GPU 130 may include hundreds or thousands of processingcores or “streaming multiprocessors” (SMs) 132 running in parallel.

In one example embodiment, the GPU 130 includes a plurality ofprogrammable high performance processors that can be referred to as“streaming multiprocessors” (“SMs”) 132, and a hardware-based graphicspipeline including a graphics primitive engine 134 and a raster engine136. These components of the GPU 130 are configured to perform real-timeimage rendering using a technique called “scan conversion rasterization”to display three-dimensional scenes on a two-dimensional display 150. Inrasterization, geometric building blocks (e.g., points, lines,triangles, quads, meshes, etc.) of a 3D scene are mapped to pixels ofthe display (often via a frame buffer memory).

The GPU 130 converts the geometric building blocks (i.e., polygonprimitives such as triangles) of the 3D model into pixels of the 2Dimage and assigns an initial color value for each pixel. The graphicspipeline may apply shading, transparency, texture and/or color effectsto portions of the image by defining or adjusting the color values ofthe pixels. The final pixel values may be anti-aliased, filtered andprovided to the display 150 for display. Many software and hardwareadvances over the years have improved subjective image quality usingrasterization techniques at frame rates needed for real-time graphics(i.e., 30 to 60 frames per second) at high display resolutions such as4096×2160 pixels or more on one or multiple displays 150.

To enable the GPU 130 to perform ray tracing in real time in anefficient manner, the GPU provides one or more “TTUs” 138 coupled to oneor more SMs 132. The TTU 138 includes hardware components configured toperform (or accelerate) operations commonly utilized in ray tracingalgorithms A goal of the TTU 138 is to accelerate operations used in raytracing to such an extent that it brings the power of ray tracing toreal-time graphics application (e.g., games), enabling high-qualityshadows, reflections, and global illumination. Results produced by theTTU 138 may be used together with or as an alternative to other graphicsrelated operations performed in the GPU 130.

More specifically, SMs 132 and the TTU 138 may cooperate to cast raysinto a 3D model and determine whether and where that ray intersects themodel's geometry. Ray tracing directly simulates light traveling througha virtual environment or scene. The results of the ray intersectionstogether with surface texture, viewing direction, and/or lightingconditions are used to determine pixel color values. Ray tracingperformed by SMs 132 working with TTU 138 allows for computer-generatedimages to capture shadows, reflections, and refractions in ways that canbe indistinguishable from photographs or video of the real world. Sinceray tracing techniques are even more computationally intensive thanrasterization due in part to the large number of rays that need to betraced, the TTU 138 is capable of accelerating in hardware certain ofthe more computationally-intensive aspects of that process.

Given a BVH constructed as described above, the TTU 138 performs a treesearch where each node in the tree visited by the ray has a boundingvolume for each descendent branch or leaf, and the ray only visits thedescendent branches or leaves whose corresponding bound volume itintersects. In this way, TTU 138 explicitly tests only a small number ofprimitives for intersection, namely those that reside in leaf nodesintersected by the ray. In the example non-limiting embodiments, the TTU138 accelerates both tree traversal (including the ray-volume tests) andray-primitive tests. As part of traversal, it can also handle at leastone level of instance transforms, transforming a ray from world-spacecoordinates into the coordinate system of an instanced mesh. In theexample non-limiting embodiments, the TTU 138 does all of this in MIMDfashion, meaning that rays are handled independently once inside theTTU.

In the example non-limiting embodiments, the TTU 138 operates as aservant (coprocessor) to the SMs (streaming multiprocessors) 132. Inother words, the TTU 138 in example non-limiting embodiments does notoperate independently, but instead follows the commands of the SMs 132to perform certain computationally-intensive ray tracing related tasksmuch more efficiently than the SMs 132 could perform themselves. Inother embodiments or architectures, the TTU 138 could have more or lessautonomy.

In the examples shown, the TTU 138 receives commands via SM 132instructions and writes results back to an SM register file. For manycommon use cases (e.g., opaque triangles with at most one level ofinstancing), the TTU 138 can service the ray tracing query withoutfurther interaction with the SM 132. More complicated queries (e.g.,involving alpha-tested triangles, primitives other than triangles, ormultiple levels of instancing) may require multiple round trips(although the technology herein reduces the need for such “round trips”for certain kinds of geometry by providing the TTU 138 with enhancedcapabilities to autonomously perform multi-box ray-bounding-volumeintersection testing without the need to ask the calling SM for help).In addition to tracing rays, the TTU 138 is capable of performing moregeneral spatial queries where an AABB or the extruded volume between twoAABBs (which we call a “beam”) takes the place of the ray. Thus, whilethe TTU 138 is especially adapted to accelerate ray tracing relatedtasks, it can also be used to perform tasks other than ray tracing.

The TTU 138 thus autonomously performs a test of each ray against a widerange of bounding volumes, and can cull any bounding volumes that don'tintersect with that ray. Starting at a root node that bounds everythingin the scene, the traversal co-processor tests each ray against smaller(potentially overlapping) child bounding volumes which in turn bound thedescendent branches of the BVH. The ray follows the child pointers forthe bounding volumes the ray hits to other nodes until the leaves orterminal nodes (volumes) of the BVH are reached. As described herein,any such bounding volume can be explicated as a plurality of boundingboxes to provide a more efficient intersection test for certain geometrye.g., that cannot be convenient enclosed by axis-aligned-bounding boxes(AABBs).

Once the TTU 138 traverses the acceleration data structure to reach aterminal or “leaf” node (which may be represented by one or multiplebounding volumes) that intersects the ray and contains a geometricprimitive, it performs an accelerated ray-primitive intersection test todetermine whether the ray intersects that primitive (and thus the objectsurface that primitive defines). The ray-primitive test can provideadditional information about primitives the ray intersects that can beused to determine the material properties of the surface required forshading and visualization. Recursive traversal through the accelerationdata structure enables the traversal co-processor to discover all objectprimitives the ray intersects, or the closest (from the perspective ofthe viewpoint) primitive the ray intersects (which in some cases is theonly primitive that is visible from the viewpoint along the ray). Seee.g., Lefrancois et al, NVIDIA Vulkan Ray Tracing Tutorial, December2019, https://developer.nvidia.com/rtx/raytracing/vkray

As mentioned above, the TTU 138 also accelerates the transform of eachray from world space into object space to obtain finer and finerbounding box encapsulations of the primitives and reduce the duplicationof those primitives across the scene. As described above, objectsreplicated many times in the scene at different positions, orientationsand scales can be represented in the scene as instance nodes whichassociate a bounding box and leaf node in the world space BVH with atransformation that can be applied to the world-space ray to transformit into an object coordinate space, and a pointer to an object-spaceBVH. This avoids replicating the object space BVH data multiple times inworld space, saving memory and associated memory accesses. The instancetransform increases efficiency by transforming the ray into object spaceinstead of requiring the geometry or the bounding volume hierarchy to betransformed into world (ray) space and is also compatible withadditional, conventional rasterization processes that graphicsprocessing performs to visualize the primitives. However, as explainedabove in connection with FIGS. 4A-4C, technology herein reduces the needfor such efficient transforms to further increase performance.

Example Ray Tracing Processes

FIG. 10 shows an exemplary ray tracing shading pipeline 900 that may beperformed by SM 132 and accelerated by TTU 138. The ray tracing shadingpipeline 900 starts by an SM 132 invoking ray generation 910 and issuinga corresponding ray tracing request to the TTU 138. The ray tracingrequest identifies a single ray cast into the scene and asks the TTU 138to search for intersections with an acceleration data structure the SM132 also specifies. The TTU 138 traverses (FIG. 10 block 920) theacceleration data structure to determine intersections or potentialintersections between the ray and the volumetric subdivisions andassociated triangles the acceleration data structure represents.Potential intersections can be identified by finding bounding volumes inthe acceleration data structure that are intersected by the ray.Descendants of non-intersected bounding volumes need not be examined.

For triangles within intersected bounding volumes, the TTU 138ray-primitive test block 720 performs an intersection 930 process todetermine whether the ray intersects the primitives. The TTU 138 returnsintersection information to the SM 132, which may perform an “any hit”shading operation 940 in response to the intersection determination. Forexample, the SM 132 may perform (or have other hardware perform) atexture lookup for an intersected primitive and decide based on theappropriate texel's value how to shade a pixel visualizing the ray. TheSM 132 keeps track of such results since the TTU 138 may return multipleintersections with different geometry in the scene in arbitrary order.

FIG. 11 is a flowchart summarizing example ray tracing operations theTTU 138 performs as described above in cooperation with SM(s) 132. TheFIG. 11 operations are performed by TTU 138 in cooperation with itsinteraction with an SM 132. The TTU 138 may thus receive theidentification of a ray from the SM 132 and traversal state enumeratingone or more nodes in one or more BVH's that the ray must traverse. TheTTU 138 determines which bounding volumes of a BVH data structure theray intersects (the “ray-complet” test 512). The TTU 138 can alsosubsequently determine whether the ray intersects one or more primitivesin the intersected bounding volumes and which triangles are intersected(the “ray-primitive test” 520)—or the SM 132 can perform this test insoftware if it is too complicated for the TTU to perform itself. Inexample non-limiting embodiments, complets specify root or interiornodes (i.e., volumes) of the bounding volume hierarchy with childrenthat are other complets or leaf nodes of a single type per complet.

First, the TTU 138 inspects the traversal state of the ray. If a stackthe TTU 138 maintains for the ray is empty, then traversal is complete.If there is an entry on the top of the stack, the traversal co-processor138 issues a request to the memory subsystem to retrieve that node. Thetraversal co-processor 138 then performs a bounding box test 512 todetermine if a bounding volume of a BVH data structure is intersected bya particular ray the SM 132 specifies (step 512, 514). In examplenon-limiting embodiments herein, this bounding box test 512 is modifiedto include a plurality of intersection tests, one or each of a pluralityof multi-boxes uses in combo to conservative enclose the geometricprimitive(s). If the bounding box test determines that the boundingvolume is not intersected by the ray (“No” in step 514), then there isno need to perform any further testing for visualization and the TTU 138can return this result to the requesting SM 132. This is because if aray misses a bounding volume (as in FIG. 1A with respect to boundingvolume 310), then the ray will miss all other smaller bounding volumesinside the bounding volume being tested and any primitives that boundingvolume contains.

If the bounding box test performed by the TTU 138 reveals that thebounding volume is intersected by the ray (“Yes” in Step 514), then theTTU determines if the bounding volume can be subdivided into smallerbounding volumes (step 518). In one example embodiment, the TTU 138isn't necessarily performing any subdivision itself. Rather, each nodein the BVH has one or more children (where each child is a leaf or abranch in the BVH). For each child, there is one or more boundingvolumes and a pointer that leads to a branch or a leaf node. When a rayprocesses a node using TTU 138, it is testing itself against thebounding volumes of the node's children. The ray only pushes stackentries onto its stack for those branches or leaves whose representativebounding volumes were hit. When a ray fetches a node in the exampleembodiment, it doesn't test against the bounding volume of the node—ittests against the bounding volumes of the node's children. The TTU 138pushes nodes whose bounding volumes are hit by a ray onto the ray'straversal stack in an order determined by ray configuration. Forexample, it is possible to push nodes onto the traversal stack in theorder the nodes appear in memory, or in the order that they appear alongthe length of the ray, or in some other order. If there are furthersubdivisions of the bounding volume (“Yes” in step 518), then thosefurther subdivisions of the bounding volume are accessed and thebounding box test is performed for each of the resulting subdividedbounding volumes to determine which subdivided bounding volumes areintersected by the ray and which are not. In this recursive process,some of the bounding volumes may be eliminated by test 514 while otherbounding volumes may result in still further and further subdivisionsbeing tested for intersection by TTU 138 recursively applying steps512-518.

Once the TTU 138 determines that the bounding volumes intersected by theray are leaf nodes (“No” in step 518), the TTU 138 and/or SM 132performs a primitive (e.g., triangle) intersection test 520 to determinewhether the ray intersects primitives in the intersected boundingvolumes and which primitives the ray intersects. The TTU 138 thusperforms a depth-first traversal of intersected descendent branch nodesuntil leaf nodes are reached. The TTU 138 processes the leaf nodes. Ifthe leaf nodes are primitive ranges, the TTU 138 or the SM 132 teststhem against the ray. If the leaf nodes are instance nodes, the TTU 138or the SM 132 applies the instance transform. If the leaf nodes are itemranges, the TTU 138 returns them to the requesting SM 132. In theexample non-limiting embodiments, the SM 132 can command the TTU 138 toperform different kinds of ray-primitive intersection tests and reportdifferent results depending on the operations coming from an application(or an software stack the application is running on) and relayed by theSM to the TTU. For example, the SM 132 can command the TTU 138 to reportthe nearest visible primitive revealed by the intersection test, or toreport all primitives the ray intersects irrespective of whether theyare the nearest visible primitive. The SM 132 can use these differentresults for different kinds of visualization. Or the SM 132 can performthe ray-primitive intersection test itself once the TTU 138 has reportedthe ray-complet test results. Once the TTU 138 is done processing theleaf nodes, there may be other branch nodes (pushed earlier onto theray's stack) to test.

Example Non-Limiting TTU 138 Hardware Implementation

FIG. 12 shows an example simplified block diagram of TTU 138 includinghardware configured to perform accelerated traversal operations asdescribed above. In some embodiments, the TTU 138 may perform adepth-first traversal of a bounding volume hierarchy using a short stacktraversal with intersection testing of supported leaf node primitivesand mid-traversal return of alpha primitives and unsupported leaf nodeprimitives (items). The TTU 138 includes dedicated hardware to determinewhether a ray intersects bounding volumes and dedicated hardware todetermine whether a ray intersects primitives of the tree datastructure.

In more detail, TTU 138 includes an intersection management block 722, aray management block 730 and a stack management block 740. Each of theseblocks (and all of the other blocks in FIG. 12) may constitute dedicatedhardware implemented by logic gates, registers, hardware-embedded lookuptables or other combinatorial logic, etc.

The ray management block 730 is responsible for managing informationabout and performing operations concerning a ray specified by an SM 132to the ray management block. The stack management block 740 works inconjunction with traversal logic 712 to manage information about andperform operations related to traversal of a BVH acceleration datastructure. Traversal logic 712 is directed by results of a ray-complettest block 710 that tests intersections between the ray indicated by theray management block 730 and volumetric subdivisions represented by theBVH, using instance transforms as needed. The ray-complet test block 710retrieves additional information concerning the BVH from memory 140 viaan L0 complet cache 752 that is part of the TTU 138. The results of theray-complet test block 710 informs the traversal logic 712 as to whetherfurther recursive traversals are needed. The stack management block 740maintains stacks to keep track of state information as the traversallogic 712 traverses from one level of the BVH to another, with the stackmanagement block 740 pushing items onto the stack as the traversal logictraverses deeper into the BVH and popping items from the stack as thetraversal logic traverses upwards in the BVH. The stack management block740 is able to provide state information (e.g., intermediate or finalresults) to the requesting SM 132 at any time the SM requests.

The intersection management block 722 manages information about andperforms operations concerning intersections between rays andprimitives, using instance transforms as needed. The ray-primitive testblock 720 retrieves information concerning geometry from memory 140 onan as-needed basis via an L0 primitive cache 754 that is part of TTU138. The intersection management block 722 is informed by results ofintersection tests the ray-primitive test and transform block 720performs. Thus, the ray-primitive test and transform block 720 providesintersection results to the intersection management block 722, whichreports geometry hits and intersections to the requesting SM 132.

A Stack Management Unit 740 inspects the traversal state to determinewhat type of data needs to be retrieved and which data path (complet orprimitive) will consume it. The intersections for the bounding volumesare determined in the ray-complet test path of the TTU 138 including oneor more ray-complet test blocks 710 and one or more traversal logicblocks 712. A complet specifies root or interior nodes of a boundingvolume. Thus, a complet may define one or more bounding volumes for theray-complet test. In example embodiments herein, a complet may define aplurality of “child” bounding volumes that (whether or not theyrepresent leaf nodes) that don't necessarily each have descendants butwhich the TTU will test in parallel for ray-bounding volume intersectionto determine whether geometric primitives associated with the pluralityof bounding volumes need to be tested for intersection.

The ray-complet test path of the TTU 138 identifies which boundingvolumes are intersected by the ray. Bounding volumes intersected by theray need to be further processed to determine if the primitivesassociated with the intersected bounding volumes are intersected. Theintersections for the primitives are determined in the ray-primitivetest path including one or more ray-primitive test and transform blocks720 and one or more intersection management blocks 722.

The TTU 138 receives queries from one or more SMs 132 to perform treetraversal operations. The query may request whether a ray intersectsbounding volumes and/or primitives in a BVH data structure. The querymay identify a ray (e.g., origin, direction, and length of the ray) anda BVH data structure and traversal state (short stack) which includesone or more entries referencing nodes in one or more Bounding VolumeHierarchies that the ray is to visit. The query may also includeinformation for how the ray is to handle specific types of intersectionsduring traversal. The ray information may be stored in the raymanagement block 730. The stored ray information (e.g., ray length) maybe updated based on the results of the ray-primitive test.

The TTU 138 may request the BVH data structure identified in the queryto be retrieved from memory outside of the TTU 138. Retrieved portionsof the BVH data structure may be cached in the level-zero (L0) cache 750within the TTU 138 so the information is available for othertime-coherent TTU operations, thereby reducing memory 140 accesses.Portions of the BVH data structure needed for the ray-complet test maybe stored in a L0 complet cache 752 and portions of the BVH datastructure needed for the ray-primitive test may be stored in an L0primitive cache 754.

After the complet information needed for a requested traversal step isavailable in the complet cache 752, the ray-complet test block 710determines bounding volumes intersected by the ray. In performing thistest, the ray may be transformed from the coordinate space of thebounding volume hierarchy to a coordinate space defined relative to acomplet. The ray is tested against the bounding boxes associated withthe child nodes of the complet. In the example non-limiting embodiment,the ray is not tested against the complet's own bounding box because (1)the TTU 138 previously tested the ray against a similar bounding boxwhen it tested the parent bounding box child that referenced thiscomplet, and (2) a purpose of the complet bounding box is to define alocal coordinate system within which the child bounding boxes can beexpressed in compressed form. If the ray intersects any of the childbounding boxes, the results are pushed to the traversal logic todetermine the order that the corresponding child pointers will be pushedonto the traversal stack (further testing will likely require thetraversal logic 712 to traverse down to the next level of the BVH).These steps are repeated recursively until intersected leaf nodes of theBVH are encountered

The ray-complet test block 710 may provide ray-complet intersections tothe traversal logic 712. Using the results of the ray-complet test, thetraversal logic 712 creates stack entries to be pushed to the stackmanagement block 740. The stack entries may indicate internal nodes(i.e., a node that includes one or more child nodes) that need to befurther tested for ray intersections by the ray-complet test block 710and/or triangles identified in an intersected leaf node that need to betested for ray intersections by the ray-primitive test and transformblock 720. The ray-complet test block 710 may repeat the traversal oninternal nodes identified in the stack to determine all leaf nodes inthe BVH that the ray intersects. The precise tests the ray-complet testblock 710 performs will in the example non-limiting embodiment bedetermined by mode bits, ray operations (see below) and culling of hits,and the TTU 138 may return intermediate as well as final results to theSM 132.

Multi-Box Implementation

FIG. 14 is a flowchart of an example non-limiting method for acceleratedray-bounding volume intersection test performed by TTU 138 usingmultibox techniques. The method may be performed by a TTU 138 (e.g.,ray-complet test unit 710 and traversal logic 712 of the TTU 138)disclosed in this application, but is not so limited.

The method includes receiving a request for a complet intersection test402. The complet intersection test may include identifying intersectionsbetween a query data structure (e.g., a ray) and one or more child nodes(each identified by one or more bounding volumes) identified in acomplet 402. The request may be received from an SM or may be based onprevious results of ray-complet test performed by the ray-complet testpath of the TTU. In some embodiments, the query data structure may be aray given by its three-coordinate origin, three-coordinate direction,and/or minimum and maximum values for the t-parameter along the ray. Thecomplet may be identified by location of complet in memory or pointer tonode of a BVH structure, which may be an inner node or a leaf node.

In response to the request, the complet is retrieved from memory 404.The complet may be retrieved from the TTU memory (e.g., complet cache752 of L0 cache 750) or memory outside of the TTU. The complet may forexample be provided as a single cacheline-sized block (e.g., 128B) or ina contiguous group of cacheline-sized blocks. The cacheline-sized blockmay include header information and bounding volume information. See FIG.8 for example formats of a complet. Each cacheline-sized block mayinclude a predetermined number of sub-blocks, each sub-blockcorresponding to a child node of the current node.

The header information of the block may include type of geometry withinthe block, node type information (e.g., inner node or leaf node), numberof child nodes in the block, number of bodes in each child node, and/orone or more override flags for controlling how ray determined tointersect a bounding volume is treated (e.g., with or without cullingthe ray). See FIG. 8.

The child node information may identify one or more bounding volumes foreach child node represented by the block. In the typical case there willbe a single AABB box for each child node, but in a multi-box case therewill be more than one AABB box for each child node represented by theblock. Accordingly, a child may be associated with a single boundingvolume (e.g., a box) identified in the child node information or aplurality of bounding volumes (e.g., sub-box or sub-volume). Forexample, each child node may be identified with two sub-boxes, threesub-boxes, or twelve sub-boxes. In some examples, the child nodeinformation may include each child node including the same number ofsub-boxes. In other examples, the child node information may includechild nodes identified with a mixed number of boxes. In one example, oneor more child nodes may be identified with a single bounding volume(e.g., a box) and one or more other child nodes may be identified with aplurality of boxes. However, in the FIG. 11 example formats, all childnodes are associated with the same number of bounding volumes (e.g., 1,2, 3, 4, 6 or 12).

The method includes determining if the complet identifies a multi-boxchild 406. The determination may be made based on a flag set in theheader of the complet and/or for each child identified in the complet.

When the determination is made that the complet does not identify amulti-box child (No in step 406), the method includes determiningwhether the single bounding volume associated with each child in thecomplet is intersected by the ray 408. This ray-complet intersectiontest may determine whether the ray intersects an area identified by thechild's bounding volume (e.g., defined by vertices of a box inobject-space coordinates). To test the ray against the area identifiedby the child's bounding volume, the ray may be transformed into theobject-space coordinates of the bounding volume using instancetransforms.

The method includes returning an intersection result identifying eachchild bounding volume that was intersected by the ray 410. The resultsmay be returned to the SM, the traversal logic 712 and/or the stackmanagement unit 740 for performing further tests in the ray-primitivetest path and/or the ray-complet test path of the TTU based on theresults. The intersection results may include ray intersection position.For example, the intersection results may include a t-min value for theray indicating a position along the ray where the ray intersects thechild's bounding volume. The respective ray intersection position may beprovided for each child determined to be intersected by the ray.

When the determination is made that the complet includes a multi-boxchild (YES in step 406), the method includes testing all of pluralbounding volumes associated with the ray to determine whether any areintersected by ray 412 Similar to step 408, identifying boxes associatedwith the child that are intersected by the ray may include determiningwhether the ray intersects an area identified by the bounding volumes ofany of the multi-boxes (e.g., by vertices of a box in object-spacecoordinates). To test the ray against the areas identified by themulti-boxes, the ray may be transformed into the object-spacecoordinates of the bounding volumes using instance transforms.

The method includes returning intersection results identifying eachchild multi-box intersected by the ray 414. Operations 412 and 414 mayreport each (sub)bounding volume determined to be intersected by the rayto the transversal logic 712, which may in turn OR together all suchintersection results for all multiboxes of a child to determine a singleBoolean True/False value indicative of intersection/no intersection.

The results may be returned to the SM, the traversal logic 712 and/orthe stack management unit 740 for performing further tests in theray-primitive test path and/or the ray-complet test path of the TTUbased on the results. The intersection results may include informationon intersection position of the ray. For example, the intersectionresults may include a lowest t-min value for all intersected boxes inthe multi-box bounding volume.

Example Hardware Implementation of RCT 710 and TL 712

The multi-box mechanism is implemented efficiently in the ray-complettest (RCT) 710 and traversal logic (TL) 712 portions of the TTU 138 ofFIGS. 9 & 12. On the surface, it is simply an OR of box intersectionsfor multiple boxes in a set into the hit result of a single child, whilealso finding the lowest t-min intersection of all the boxes in the setfort reporting purposes (see FIG. 9). In RCT 710, all boxes are operatedon in individual lanes 1202 just as if there were no multi-box. Thesettings for multi-box are decoded in RCT 710 and passed to TL 712.

FIG. 15 illustrates a hardware that may perform the ray-completintersection test according to an exemplary embodiment of thisdisclosure. The hardware circuitry may be included in the TTU 138. Thehardware circuitry may include ray-complet intersection test circuitry1202, t-min comparator circuitry 1204, and a logic network 1206 (e.g.,OR Gate network). The hardware circuitry shown in FIG. 9 may receive rayinformation and a complet and provide a single hit result and t-minvalue for each child. The complet may be provide in a multi-box completformat. In some examples, at least a portion of the hardware circuitryfor performing the ray-complet intersection test may be included inand/or between the ray-complet test block 710 and/or traversal logic712.

For correct reporting of t-min value, the reported t-min should be theminimum of all t-min values for all intersected boxes. FIG. 16 shows anexample such t-min comparator circuit 1204. In the example non-limitingembodiment, the t-min comparator network 1204 comes at the end of theRCT 710 pipeline before the t-min value is generated for each box. Foreach multi-box, RCT 710 will find the minimum t-min value and swap thatvalue into the lowest ordinal child in the multi-box before presentingthe results to TL 712.

As shown in FIG. 12, ray information and a block of data (e.g., acacheline-sized block) including a header and child information areprovided to the hardware circuitry. The child information is provided ina predetermined number of sub-blocks (N+1), each sub-block identifying achild (e.g., a bounding volume of the child) or a box of a multi-boxchild (e.g., a bounding volume of the multi-box child box).

The ray information and information from each of the sub-blocks isprovided to the ray-complet intersection test circuitry 1202. Theray-complet intersection test circuitry 1202 is configured to receivethe ray information and the sub block information and determine if theray intersects a bounding volume identified in each sub block. As shownin FIG. 15, the ray-complet intersection test circuitry 1202 may includededicated circuitry associated with each sub-block identifying a childbounding volume or a box of a multi-box child, so that the boundingvolumes identified in each sub-block can be processed simultaneously inparallel. In other example embodiments, the intersection test for eachsub-block performed by the ray-complet intersection test circuitry 1202can be performed in series, in a sequence, in stages, or in a pipeline.In such non-parallel implementations, testing of further ones of themultiboxes associated with particular geometry could be avoided as soonas one of the multi-boxes is found to interest the ray.

For each sub-block, the ray-complet intersection test circuitry 1202 mayidentify whether there is a hit or no-hit (e.g., indicated by 0 or 1 ina hit/miss bit) and a t-min value of the intersection. The t-min valueindicates a position along the ray where the ray first intersects thebounding volume from the ray origin.

The logic network 1206 is configured to receive the hit/no-hit resultsfor each sub-block and return a single hit result for each child. Thelogic network 1206 (which may be part of TL 712) is configured to returna single hit result for a child even if a plurality of boxes of amulti-box child are determined to be intersected by the ray. In the casewhere each sub-block in the block identifies a bounding volume, thelogic network 1206 may simply pass through the results for each subblock. In the case where two or more sub-blocks identify boundingvolumes of a same child node, the logic network 1206 may return a singlehit if at least one bounding volume of the same child node is determinedto be intersected by the ray.

The logic network 1206 may receive information from the headeridentifying a number of child nodes in the block of data and/or a numberof multi-boxes in each child node identified in the block of data. Basedon this information, the logic network 1206 is configured to group thehit/no-hit results such that a hit is provided for each child when atleast one multi-box in a child is determined to be intersected by theray.

The logic network 1206 may include an OR gate (or NOR gate) network thatis configured based on the number of child nodes in the block and/or anumber of multi-boxes in each child node. For example, in the case whereeach child is identified with two sub-blocks (each sub-block identifyinga bounding volume of the same child node), an OR (NOR) gate may receivehit/no-hit results from two intersection test circuits and return a hitfor the child when one or both of the two intersection test circuitsindicates a hit for the tested bounding volume.

The t-min comparator circuitry 1204 is configured to receive a t-minvalue from each bounding volume determined to be intersected by the ray,and return a t-min value for each child Similar to the logic network1206, the t-min comparator circuitry 1204 may receive information fromthe header identifying a number of child nodes in the block of dataand/or a number of multi-boxes in each child node identified in theblock of data. Based on this information, the t-min comparator circuitry1204 is configured to determine a minimum of all t-min value for allintersected boxes of a child. In one example, the t-min comparatorcircuitry 1204 is configured to find, for each box of a multi-boxcomplet, the minimum t-min value and swap that value into the lowestordinal child in the multi-box before presenting the results.

The t-min comparator circuitry 1204 reporting the minimum t-min valuefor the child provides for the ray shortening to a closest rayintersection point from the ray origin for each bounding volumesidentified by boxes in the multi-box child. While including the t-mincomparator circuitry 1204 may add an additional pipestage to theray-complet test path of the TTU in one embodiment, shortening thelength of the ray based on the t-min value may reduce computationsneeded when further child nodes of the intersected child node and/orprimitives of the intersected child node are tested for intersection bythe ray.

In the above examples, the hardware is configured to return ray-completintersection results without reporting each intersected bounding volumeidentified in the block and associated with a child node and/or eacht-min value of the intersections for the child node. As discussed above,the single hit or no hit is provided for each child node even when aplurality of bounding volumes associated with the child are intersected,and a minimum t-min value from a plurality of t-min values is providedfor each intersected child node.

For correct reporting of t-min value (which in example embodiments isused to determine which geometry to check first when performing theray-primitive intersection test, and may also be used to performfiltering based on tmin), the reported t-min should be the minimum ofall t-min values for all intersected boxes. The t-min comparator networkcomes at the end of the ray-complet test pipeline before the t-min valueis generated for each box. For each multi-box, ray-complet test willfind the minimum t-min value and swap that value into the lowest ordinalchild in the multi-box before presenting the results to TL.

Table 1 and FIG. 16 (please also refer to FIG. 8) show an examplenon-limiting layout of the FIG. 12 comparator network 1204 for a blockincluding twelve sub-blocks where, e.g., [0,1] is a comparison andselect between t-min of child 0 and 1 whose output is ‘01’ and is usedin lower layers:

TABLE 1 Layer 1 (2 box answer): [0, 1] [2, 3] [4, 5] [6, 7] [8, 9[ [A,B] Layer 2a (3 box answer): [01, 2] [3, 45] [67, 8] [9, AB] Layer 2b (4box answer): [01, 23] [45, 67] [89, AB] Layer 3 (6 box answer): [0123,45] [67, 89AB] Layer 4 (12 box answer): [012345, 6789AB]

An example non-limiting layout of the t-min comparator network 1204,e.g., [0,1] provides a comparison and select between t-min of child 0and 1 whose output is ‘01’ and is used in lower layers. Making this lesscostly in some example embodiments is that all t-min values produced byRCT 710 have a common exponent. That means the comparison works only onmantissas.

FIG. 17 shows example pseudo-code for one example implementation of thebeginning of a process performed by TL 712. As shown, at the beginningof TL 712, the resulting hit/no-hit are ORed together. This functionrepresented by an OR (or NOR) gate network 1206 shown in FIG. 15 is partof the TL 712. The output of OR (NOR) gate network 1206 is used todetermine whether any of the multi-boxes are found to intersect the ray.If any box in a set is intersected, then that child is treated as a hit.As FIG. 17 illustrates, there are many different possibleimplementations for this functionality including OR NOR gates, one ormore lookup tables, a pipelined/staged hardware state machine, etc.

In example non-limiting embodiments, Ray Ops are valid for only theprimary (numerically lowest) child in a set of explicated plurality ofbounding volumes. All other Ray Ops are unused and reserved at thistime. However, in the future, it may be possible to perform differentray operations on different ones of these bounding volumes e.g., togenerate information additional to intersection.

Ray-Primitive Intersection Testing

Referring again to FIG. 12, the TTU 138 also has the ability toaccelerate intersection tests that determine whether a ray intersectsparticular geometry or primitives enclosed by multi-box boundingvolumes. For some cases in which multi-box is used, the geometry issufficiently complex (e.g., defined by curves or other abstractconstructs as opposed to e.g., vertices) that TTU 138 in someembodiments may not be able to help with the ray-primitive intersectiontesting. In such cases, the TTU 138 simply reports the ray-completintersection test results to the SM 132, and the SM 132 performs theray-primitive intersection test itself. In other cases (e.g.,triangles), the TTU 138 can perform the ray-triangle intersection testitself, thereby further increasing performance of the overall raytracing process. For sake of completeness, the following describes howthe TTU 138 can perform or accelerate the ray-primitive intersectiontesting.

As explained above, leaf nodes (and in particular, multi-boxesassociated with such leaf nodes) found to be intersected by the rayidentify (enclose) primitives that may or may not be intersected by theray. One option is for the TTU 138 to provide e.g., a range of geometryidentified in the intersected leaf nodes to the SM 132 for furtherprocessing. For example, the SM 132 may itself determine whether theidentified primitives are intersected by the ray based on theinformation the TTU 138 provides as a result of the TTU traversing theBVH. To offload this processing from the SM 132 and thereby accelerateit using the hardware of the TTU 138, the stack management block 740 mayissue requests for the ray-primitive and transform block 720 to performa ray-primitive test for the primitives within intersected leaf nodesthe TTU's ray-complet test block 710 identified. In some embodiments,the SM 132 may issue a request for the ray-primitive test to test aspecific range of primitives and transform block 720 irrespective of howthat geometry range was identified.

After making sure the primitive data needed for a requestedray-primitive test is available in the primitive cache 754, theray-primitive and transform block 720 may determine primitives that areintersected by the ray using the ray information stored in the raymanagement block 730. The ray-primitive test block 720 provides theidentification of primitives determined to be intersected by the ray tothe intersection management block 722.

The intersection management block 722 can return the results of theray-primitive test to the SM 132. The results of the ray-primitive testmay include identifiers of intersected primitives, the distance ofintersections from the ray origin and other information concerningproperties of the intersected primitives. In some embodiments, theintersection management block 722 may modify an existing ray-primitivetest (e.g., by modifying the length of the ray) based on previousintersection results from the ray-primitive and transform block 720.

The intersection management block 722 may also keep track of differenttypes of primitives. For example, the different types of trianglesinclude opaque triangles that will block a ray when intersected andalpha triangles that may or may not block the ray when intersected ormay require additional handling by the SM. Whether a ray is blocked ornot by a transparent triangle may for example depend on texture(s)mapped onto the triangle, area of the triangle occupied by the textureand the way the texture modifies the triangle. For example, transparency(e.g., stained glass) in some embodiments requires the SM 132 to keeptrack of transparent object hits so they can be sorted and shaded inray-parametric order, and typically don't actually block the ray.Meanwhile, alpha “trimming” allows the shape of the primitive to betrimmed based on the shape of a texture mapped onto the primitive—forexample, cutting a leaf shape out of a triangle. (Note that in rastergraphics, transparency is often called “alpha blending” and trimming iscalled “alpha test”). In other embodiments, the TTU 138 can pushtransparent hits to queues in memory for later handling by the SM 132and directly handle trimmed triangles by sending requests to the textureunit. Each triangle may include a designator to indicate the triangletype. The intersection management block 722 is configured to maintain aresult queue for tracking the different types of intersected triangles.For example, the result queue may store one or more intersected opaquetriangle identifiers in one queue and one or more transparent triangleidentifiers in another queue.

For opaque triangles, the ray intersection for less complex geometry canbe fully determined in the TTU 138 because the area of the opaquetriangle blocks the ray from going past the surface of the triangle. Fortransparent triangles, ray intersections cannot in some embodiments befully determined in the TTU 138 because TTU 138 performs theintersection test based on the geometry of the triangle and may not haveaccess to the texture of the triangle and/or area of the triangleoccupied by the texture (in other embodiments, the TTU may be providedwith texture information by the texture mapping block of the graphicspipeline). To fully determine whether the triangle is intersected,information about transparent triangles the ray-primitive and transformblock 720 determines are intersected may be sent to the SM 132, for theSM to make the full determination as to whether the triangle affectsvisibility along the ray.

The SM 132 can resolve whether or not the ray intersects a textureassociated with the transparent triangle and/or whether the ray will beblocked by the texture. The SM 132 may in some cases send a modifiedquery to the TTU 138 (e.g., shortening the ray if the ray is blocked bythe texture) based on this determination. In one embodiment, the TTU 138may be configured to return all triangles determined to intersect theray to the SM 132 for further processing. Because returning everytriangle intersection to the SM 132 for further processing is costly interms of interface and thread synchronization, the TTU 138 may beconfigured to hide triangles which are intersected but are provablycapable of being hidden without a functional impact on the resultingscene. For example, because the TTU 138 is provided with triangle typeinformation (e.g., whether a triangle is opaque or transparent), the TTU138 may use the triangle type information to determine intersectedtriangles that are occluded along the ray by another intersecting opaquetriangle and which thus need not be included in the results because theywill not affect the visibility along the ray. If the TTU 138 knows thata triangle is occluded along the ray by an opaque triangle, the occludedtriangle can be hidden from the results without impact on visualizationof the resulting scene.

The intersection management block 722 may include a result queue forstoring hits that associate a triangle ID and information about thepoint where the ray hit the triangle. When a ray is determined tointersect an opaque triangle, the identity of the triangle and thedistance of the intersection from the ray origin can be stored in theresult queue. If the ray is determined to intersect another opaquetriangle, the other intersected opaque triangle can be omitted from theresult if the distance of the intersection from the ray origin isgreater than the distance of the intersected opaque triangle alreadystored in the result queue. If the distance of the intersection from theray origin is less than the distance of the intersected opaque trianglealready stored in the result queue, the other intersected opaquetriangle can replace the opaque triangle stored in the result queue.After all of the triangles of a query have been tested, the opaquetriangle information stored in the result queue and the intersectioninformation may be sent to the SM 132.

In some embodiments, once an opaque triangle intersection is identified,the intersection management block 722 may shorten the ray stored in theray management block 730 so that bounding volumes (which may includetriangles) behind the intersected opaque triangle (along the ray) willnot be identified as intersecting the ray.

The intersection management block 722 may store information aboutintersected transparent triangles in a separate queue. The storedinformation about intersected transparent triangles may be sent to theSM 132 for the SM to resolve whether or not the ray intersects a textureassociated with the triangle and/or whether the texture blocks the ray.The SM may return the results of this determination to the TTU 138and/or modify the query (e.g., shorten the ray if the ray is blocked bythe texture) based on this determination.

As discussed above, the TTU 138 allows for quick traversal of anacceleration data structure (e.g., a BVH) to determine which primitives(e.g., triangles used for generating a scene) in the data structure areintersected by a query data structure (e.g., a ray). For example, theTTU 138 may determine which triangles in the acceleration data structureare intersected by the ray and return the results to the SM 132.However, returning to the SM 132 a result on every triangle intersectionis costly in terms of interface and thread synchronization. The TTU 138provides a hardware logic configured to hide those items or triangleswhich are provably capable of being hidden without a functional impacton the resulting scene. The reduction in returns of results to the SMand synchronization steps between threads greatly improves the overallperformance of traversal. The example non-limiting embodiments of theTTU 138 disclosed in this application provides for some of theintersections to be discarded within the TTU 138 without SM 132intervention so that less intersections are returned to the SM 132 andthe SM 132 does not have to inspect all intersected triangles or itemranges.

Example Instancing Pipeline Implementation by TTU 138 and SM 132

As discussed above, one example feature of embodiments herein relates tohoisting multiboxes from lower levels to higher levels of theacceleration structure. This is commonly done in example embodimentsthrough instance transforms. For sake of completeness, the followingdescribes how TTU 138 in example embodiments performs instancing andassociated transforms.

The FIG. 18A more detailed diagram of a ray-tracing pipeline flowchartshows the data flow and interaction between components for arepresentative use case: tracing rays against a scene containinggeometric primitives, with instance transformations handled in hardware.In one example non-limiting embodiment, the ray-tracing pipeline of FIG.18A is essentially software-defined (which in example embodiments meansit is determined by the SMs 132) but makes extensive use of hardwareacceleration by TTU 138. Key components include the SM 132 (and the restof the compute pipeline), the TTU 138 (which serves as a coprocessor toSM), and the L1 cache and downstream memory system, from which the TTUfetches BVH and triangle data.

The pipeline shown in FIG. 18A shows that bounding volume hierarchycreation 1002 can be performed ahead of time by a development system. Italso shows that ray creation and distribution 1004 are performed orcontrolled by the SM 132 or other software in the example embodiment, asshading (which can include lighting and texturing). The example pipelineincludes a “top level” BVH tree traversal 1006, ray transformation 1014,“bottom level” BVH tree traversal 1018, and a ray/triangle (or otherprimitive) intersection 1026 that are each performed by the TTU 138.These do not have to be performed in the order shown, as handshakingbetween the TTU 138 and the SM 132 determines what the TTU 138 does andin what order.

The SM 132 presents one or more rays to the TTU 138 at a time. Each raythe SM 132 presents to the TTU 138 for traversal may include the ray'sgeometric parameters, traversal state, and the ray's ray flags, modeflags and ray operations information. In an example embodiment, a rayoperation (RayOp) provides or comprises an auxiliary arithmetic and/orlogical test to suppress, override, and/or allow storage of anintersection. The traversal stack may also be used by the SM 132 tocommunicate certain state information to the TTU 138 for use in thetraversal. A new ray query may be started with an explicit traversalstack. For some queries, however, a small number of stack initializersmay be provided for beginning the new query of a given type, such as,for example: traversal starting from a complet; intersection of a raywith a range of triangles; intersection of a ray with a range oftriangles, followed by traversal starting from a complet; vertex fetchfrom a triangle buffer for a given triangle, etc. In some embodiments,using stack initializers instead of explicit stack initializationimproves performance because stack initializers require fewer streamingprocessor registers and reduce the number of parameters that need to betransmitted from the streaming processor to the TTU.

In the example embodiment, a set of mode flags the SM 132 presents witheach query (e.g., ray) may at least partly control how the TTU 138 willprocess the query when the query intersects the bounding volume of aspecific type or intersects a primitive of a specific primitive type.The mode flags the SM 132 provides to the TTU 138 enable the ability bythe SM and/or the application to e.g., through a RayOp, specify anauxiliary arithmetic or logical test to suppress, override, or allowstorage of an intersection. The mode flags may for example enabletraversal behavior to be changed in accordance with such aspects as, forexample, a depth (or distance) associated with each bounding volumeand/or primitive, size of a bounding volume or primitive in relation toa distance from the origin or the ray, particular instances of anobject, etc. This capability can be used by applications to dynamicallyand/or selectively enable/disable sets of objects for intersectiontesting versus specific sets or groups of queries, for example, to allowfor different versions of models to be used when application statechanges (for example, when doors open or close) or to provide differentversions of a model which are selected as a function of the length ofthe ray to realize a form of geometric level of detail, or to allowspecific sets of objects from certain classes of rays to make somelayers visible or invisible in specific views.

In addition to the set of mode flags which may be specified separatelyfor the ray-complet intersection and for ray-primitive intersections,the ray data structure may specify other RayOp test related parameters,such as ray flags, ray parameters and a RayOp test. The ray flags can beused by the TTU 138 to control various aspects of traversal behavior,back-face culling, and handling of the various child node types, subjectto a pass/fail status of an optional RayOp test. RayOp tests addflexibility to the capabilities of the TTU 138, at the expense of somecomplexity. The TTU 138 reserves a “ray slot” for each active ray it isprocessing, and may store the ray flags, mode flags and/or the RayOpinformation in the corresponding ray slot buffer within the TTU duringtraversal.

In the example shown in FIG. 18A, the TTU 138 performs a top level treetraversal 1006 and a bottom level tree traversal 1018. In the exampleembodiment, the two level traversal of the BVH enables fast ray tracingresponses to dynamic scene changes. As discussed above, it is possibleto hoist multi-box complet information from the bottom level treetraversal 1018 to the top level tree traversal 1006. This enablesalignment for a single primitive to be more tightly fit in its ownobject space, and then placed in a common world space e.g., via atransform by the BVH builder while reducing the number of instancetransforms the TTU 138 needs to perform during real time ray tracingoperations.

Ray transformation 1014 provides the appropriate transition from the toplevel tree traversal 1006 to the bottom level tree traversal 1018 bytransforming the ray, which may be used in the top level traversal in afirst coordinate space (e.g., world space), to a different coordinatespace (e.g., object space) of the BVH of the bottom level traversal. Anexample BVH traversal technique using a two level traversal is describedin previous literature, see, e.g., Woop, “A Ray Tracing HardwareArchitecture for Dynamic Scenes”, Universitat des Saarlandes, 2004, butembodiments are not limited thereto.

Example Top Level Tree Traversal

The top level tree traversal 1006 by TTU 138 receives complets from theL1 cache 1012, and provides an instance to the ray transformation 1014for transformation, or a miss/end output 1013 to the SM 132 for closesthit shader 1015 processing by the SM (this block can also operaterecursively based on non-leaf nodes/no hit conditions). In the top leveltree traversal 1006, a next complet fetch step 1008 fetches the nextcomplet to be tested for ray intersection in step 1010 from the memoryand/or cache hierarchy and ray-bounding volume intersection testing isdone on the bounding volumes in the fetched complet.

As described above, an instance node connects one BVH to another BVHwhich is in a different coordinate system. When a child of theintersected bounding volume is an instance node, the ray transformation1014 is able to retrieve an appropriate transform matrix from the L1cache 1016. The TTU 138, using the appropriate transform matrix,transforms the ray to the coordinate system of the child BVH. U.S.patent application Ser. No. 14/697,480, which is already incorporated byreference, describes transformation nodes that connect a first set ofnodes in a tree to a second set of nodes where the first and second setsof nodes are in different coordinate systems. The instance nodes inexample embodiments may be similar to the transformation nodes in U.S.application Ser. No. 14/697,480. In an alternative, non-instancing modeof TTU 138 shown in FIG. 18B, the TTU does not execute a “bottom” leveltree traversal 1018 and noninstanced tree BVH traversals are performedby blocks 1008, 1010 e.g., using only one stack. The TTU 138 can switchbetween the FIG. 18A instanced operations and the FIG. 18B non-instancedoperations based on what it reads from the BVH and/or query type. Forexample, a specific query type may restrict the TTU to use just thenon-instanced operations. In such a query, any intersected instancenodes would be returned to the SM.

In some non-limiting embodiments, ray-bounding volume intersectiontesting in step 1010 is performed on each bounding volume in the fetchedcomplet before the next complet is fetched. Other embodiments may useother techniques, such as, for example, traversing the top leveltraversal BVH in a depth-first manner U.S. Pat. No. 9,582,607, alreadyincorporated by reference, describes one or more complet structures andcontents that may be used in example embodiments. U.S. Pat. No.9,582,607 also describes an example traversal of complets.

When a bounding volume is determined to be intersected by the ray, thechild bounding volumes (or references to them) of the intersectedbounding volume are kept track of for subsequent testing forintersection with the ray and for traversal. In example embodiments, oneor more stack data structures is used for keeping track of childbounding volumes to be subsequently tested for intersection with theray. In some example embodiments, a traversal stack of a small size maybe used to keep track of complets to be traversed by operation of thetop level tree traversal 1006, and primitives to be tested forintersection, and a larger local stack data structure can be used tokeep track of the traversal state in the bottom level tree traversal1018. It should be noted that in the case of internal nodes withmulti-box attributes, each of the multi-boxes do not spawn childrennodes or other descendent nodes. Rather, if a downward traversal isrequired, the larger AABB bounding volume that encompasses all of themulti-boxes would generally be the bounding volume that spawns furtherchild bounding boxes, grandchild bounding boxes, etc. Hence, in at leastsome embodiments, multi-boxes do not have child nodes—either becausethey are leaf nodes that have no children, or because they areassociated with internal nodes that use larger bounding volumes for nodeinheritance.

Example Bottom Level Tree Traversal

In the bottom level tree traversal 1018, a next complet fetch step 1022fetches the next complet to be tested for ray intersection in step 1024from the memory and/or cache hierarchy 1020 and ray-bounding volumeintersection testing is done on the bounding volumes in the fetchedcomplet. The bottom level tree traversal, as noted above, may includecomplets with bounding volumes in a different coordinate system than thebounding volumes traversed in the upper level tree traversal. The bottomlevel tree traversal also receives complets from the L1 cache and canoperate recursively or iteratively within itself based onnon-leaf/no-hit conditions and also with the top level tree traversal1006 based on miss/end detection. Intersections of the ray with thebounding volumes in the lower level BVH may be determined with the raytransformed to the coordinate system of the lower level completretrieved. The leaf bounding volumes found to be intersected by the rayin the lower level tree traversal are then provided to the ray/triangleintersection 1026.

The leaf outputs of the bottom level tree traversal 1018 are provided tothe ray/triangle intersection 1026 (which has L0 cache access as well asability to retrieve triangles via the L1 cache 1028). The L0 complet andtriangle caches may be small read-only caches internal to the TTU 138.The ray/triangle intersection 1026 may also receive leaf outputs fromthe top level tree traversal 1006 when certain leaf nodes are reachedwithout traversing an instanced BVH.

After all the primitives in the primitive range have been processed, theIntersection Management Unit inspects the state of the result Queue andcrafts packets to send to the Stack Management Unit and/or RayManagement Unit to update the ray's attributes and traversal state, setup the ray's next traversal step, and/or return the ray to the SM 132(if necessary). If the result queue contains opaque or alphaintersections found during the processing of the primitive range thenthe Intersection Management Unit signals the parametric length (t) ofthe nearest opaque intersection in the result queue to the raymanagement unit to record as the ray's tmax to shorten the ray. Toupdate the traversal state to set up the ray's next traversal step theIntersection Management Unit signals to the Stack Management Unitwhether an opaque intersection from the primitive range is present inthe resultQueue, whether one or more alpha intersections are present inthe result queue, whether the resultQueue is full, whether additionalalpha intersections were found in the primitive range that have not beenreturned to the SM and which are not present in the resultQueue, and theindex of the next alpha primitive in the primitive range for the ray totest after the SM consumes the contents of the resultQueue (the index ofthe next primitive in the range after the alpha primitive with thehighest memory-order from the current primitive range in the resultqueue).

When the Stack Management Unit 740 receives the packet from IntersectionManagement Unit 722, the Stack Management Unit 740 inspects the packetto determine the next action required to complete the traversal step andstart the next one. If the packet from Intersection Management Unit 722indicates an opaque intersection has been found in the primitive rangeand the ray mode bits indicate the ray is to finish traversal once anyintersection has been found the Stack Management Unit 740 returns theray and its results queue to the SM with traversal state indicating thattraversal is complete (a done flag set and/or an empty top level andbottom level stack). If the packet from Intersection Management Unit 722indicates that there are opaque or alpha intersection in the resultqueue and that there are remaining alpha intersections in the primitiverange not present in the result queue that were encountered by the rayduring the processing of the primitive range that have not already beenreturned to the SM, the Stack Management Unit 740 returns the ray andthe result queue to the SM with traversal state modified to set the cullopaque bit to prevent further processing of opaque primitives in theprimitive range and the primitive range starting index advanced to thefirst alpha primitive after the highest alpha primitive intersectionfrom the primitive range returned to the SM in the ray's result queue.If the packet from Intersection Management Unit 722 indicates that noopaque or alpha intersections were found when the ray processed theprimitive range the Stack Management Unit 740 pops the top of stackentry (corresponding to the finished primitive range) off the activetraversal stack. If the packet from Stack Management Unit 740 indicatesor that either there are opaque intersections in the result queue andthe ray mode bits do not indicate that the ray is to finish traversalonce any intersection has been found and/or there are alphaintersections in the result queue, but there were no remaining alphaintersections found in the primitive range not present in the resultqueue that have not already been returned to the SM, the StackManagement Unit 740 pops the top of stack entry (corresponding to thefinished primitive range) off the active traversal stack and modifiesthe contents of the result queue to indicate that all intersectionspresent in the result queue come from a primitive range whose processingwas completed.

If the active stack is the bottom stack, and the bottom stack is emptythe Stack Management Unit 740 sets the active stack to the top stack. Ifthe top stack is the active stack, and the active stack is empty, thenthe Stack Management Unit 740 returns the ray and its result queue tothe SM with traversal state indicating that traversal is complete (adone flag set and/or an empty top level and bottom level stack). If theactive stack contains one or more stack entries, then the StackManagement Unit 740 inspects the top stack entry and starts the nexttraversal step. Testing of primitive and/or primitive ranges forintersections with a ray and returning results to the SM 132 aredescribed in co-pending U.S. application Ser. No. 16/101,148 entitled“Conservative Watertight Ray Triangle Intersection” and U.S. applicationSer. No. 16/101,196 entitled “Method for Handling Out-of-Order Opaqueand Alpha Ray/Primitive Intersections”, which are hereby incorporated byreference in their entireties.

Example Image Generation Pipeline Including Ray Tracing

While the above disclosure is framed in the specific context of computergraphics and visualization, ray tracing and the disclosed TTU could beused for a variety of applications beyond graphics and visualization.Non-limiting examples include sound propagation for realistic soundsynthesis, simulation of sonar systems, design of optical elements andsystems, particle transport simulation (e.g., for medical physics orexperimental high-energy physics), general wave propagation simulation,comparison to LIDAR data for purposes e.g., of robot or vehiclelocalization, and others. OptiX™ has already been used for some of theseapplication areas in the past.

For example, the ray tracing and other capabilities described above canbe used in a variety of ways. For example, in addition to being used torender a scene using ray tracing, they may be implemented in combinationwith scan conversion techniques such as in the context of scanconverting geometric building blocks (i.e., polygon primitives such astriangles) of a 3D model for generating image for display (e.g., ondisplay 150 illustrated in FIG. 9).

Meanwhile, however, the technology herein provides advantages when usedto produce images for virtual reality, augmented reality, mixed reality,video games, motion and still picture generation, and othervisualization applications. FIG. 19 illustrates an example flowchart forprocessing primitives to provide image pixel values of an image, inaccordance with an embodiment. As FIG. 19 shows, an image of a 3D modelmay be generated in response to receiving a user input (Step 1652). Theuser input may be a request to display an image or image sequence, suchas an input operation performed during interaction with an application(e.g., a game application). In response to the user input, the systemperforms scan conversion and rasterization of 3D model geometricprimitives of a scene using conventional GPU 3D graphics pipeline (Step1654). The scan conversion and rasterization of geometric primitives mayinclude for example processing primitives of the 3D model to determineimage pixel values using conventional techniques such as lighting,transforms, texture mapping, rasterization and the like as is well knownto those skilled in the art. The generated pixel data may be written toa frame buffer.

In step 1656, one or more rays may be traced from one or more points onthe rasterized primitives using TTU hardware acceleration. The rays maybe traced in accordance with the one or more ray-tracing capabilitiesdisclosed in this application. Based on the results of the ray tracing,the pixel values stored in the buffer may be modified (Step 1658).Modifying the pixel values may in some applications for example improvethe image quality by, for example, applying more realistic reflectionsand/or shadows. An image is displayed (Step 1660) using the modifiedpixel values stored in the buffer.

In one example, scan conversion and rasterization of geometricprimitives may be implemented using the processing system describedabove, and ray tracing may be implemented by the SM's 132 using the TTUarchitecture described in relation to FIG. 12, to add furthervisualization features (e.g., specular reflection, shadows, etc.). FIG.19 is just a non-limiting example—the SM's 132 could employ thedescribed TTU by itself without texture processing or other conventional3D graphics processing to produce images, or the SM's could employtexture processing and other conventional 3D graphics processing withoutthe described TTU to produce images. The SM's can also implement anydesired image generation or other functionality in software depending onthe application to provide any desired programmable functionality thatis not bound to the hardware acceleration features provided by texturemapping hardware, tree traversal hardware or other graphics pipelinehardware.

The TTU 138 in some embodiments is stateless, meaning that noarchitectural state is maintained in the TTU between queries. At thesame time, it is often useful for software running on the SM 1840 torequest continuation of a previous query, which implies that relevantstate should be written to registers by the TTU 138 and then passed backto the TTU in registers (often in-place) to continue. This state maytake the form of a traversal stack that tracks progress in the traversalof the BVH.

A small number of stack initializers may also be provided for beginninga new query of a given type, for example:

-   -   Traversal starting from a complet    -   Intersection of a ray with a range of triangles    -   Intersection of a ray with a range of triangles, followed by        traversal starting from a complet    -   Vertex fetch from a triangle buffer for a given triangle    -   Optional support for instance transforms in front of the        “traversal starting from a complet” and “intersection of a ray        with a range of triangles”.

Vertex fetch is a simple query that may be specified with request datathat consists of a stack initializer and nothing else. Other query typesmay require the specification of a ray or beam, along with the stack orstack initializer and various ray flags describing details of the query.A ray is given by its three-coordinate origin, three-coordinatedirection, and minimum and maximum values for the t-parameter along theray. A beam is additionally given by a second origin and direction.

Various ray flags can be used to control various aspects of traversalbehavior, back-face culling, and handling of the various child nodetypes, subject to a pass/fail status of an optional rayOp test. RayOpsadd considerable flexibility to the capabilities of the TTU. In someexample embodiments, the RayOps portion introduces two Ray Flag versionscan be dynamically selected based on a specified operation on dataconveyed with the ray and data stored in the complet. To explore suchflags, it's first helpful to understand the different types of childnodes allowed within a BVH, as well as the various hit types that theTTU 138 can return to the SM. Example node types are:

-   -   A child complet (i.e., an internal node) (single box or        multi-box) By default, the TTU 138 continues traversal by        descending into child complets.    -   A triangle range, corresponding to a contiguous set of triangles        within a triangle buffer    -   (1) By default, triangle ranges encountered by a ray are handled        natively by the TTU 138 by testing the triangles for        intersection and shortening the ray accordingly. If traversal        completes and a triangle was hit, default behavior is for the        triangle ID to be returned to SM 1840, along with the t-value        and barycentric coordinates of the intersection. This is the        “Triangle” hit type.    -   (2) By default, intersected triangles with the alpha bit set are        returned to SM 1840 even if traversal has not completed. The        returned traversal stack contains the state required to continue        traversal if software determines that the triangle was in fact        transparent.    -   (3) Triangle intersection in some embodiments is not supported        for beams, so encountered triangle ranges are by default        returned to SM 1840 as a “TriRange” hit type, which includes a        pointer to the first triangle block overlapping the range,        parameters specifying the range, and the t-value of the        intersection with the leaf bounding box.    -   An item range, consisting of an index (derived from a        user-provided “item range base” stored in the complet) and a        count of items.

By default, item ranges are returned to SM 1840 as an “ItemRange” hittype, consisting of for example an index, a count, and the t-value ofthe intersection with the leaf bounding box.

-   -   An instance node.

The TTU 138 in some embodiments can handle one level of instancingnatively by transforming the ray into the coordinate system of theinstance BVH. Additional levels of instancing (or every other level ofinstancing, depending on strategy) may be handled in software (or inother embodiments, the TTU 138 hardware can handle two, three or morelevels of instancing). The “InstanceNode” hit type is provided for thispurpose, consisting of a pointer to the instance node and the tvalue ofthe intersection with the leaf bounding box. In other implementations,the hardware can handle two, three or more levels of instancing.

In addition to the node-specific hit types, a generic “NodeRef” hit typeis provided that consists of a pointer to the parent complet itself, aswell as an ID indicating which child was intersected and the t-value ofthe intersection with the bounding box of that child.

An “Error” hit type may be provided for cases where the query or BVH wasimproperly formed or if traversal encountered issues during traversal.

A “None” hit type may be provided for the case where the ray or beammisses all geometry in the scene.

How the TTU handles each of the four possible node types is determinedby a set of node-specific mode flags set as part of the query for agiven ray. The “default” behavior mentioned above corresponds to thecase where the mode flags are set to all zeroes.

Alternative values for the flags allow for culling all nodes of a giventype, returning nodes of a given type to SM as a NodeRef hit type, orreturning triangle ranges or instance nodes to SM using theircorresponding hit types, rather than processing them natively within theTTU 138.

Additional mode flags may be provided for control handling of alphatriangles.

All patents & publications cited above are incorporated by reference asif expressly set forth.

While the invention has been described in connection with what ispresently considered to be the most practical and preferred embodiments,it is to be understood that the invention is not to be limited to thedisclosed embodiments, but on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the spirit andscope of the appended claims.

1. A ray tracing device comprising: memory configured to store anacceleration data structure defining geometric objects within avolumetric hierarchy; and circuitry operatively coupled to the memoryconfigured to perform operations comprising: receive information about aray; retrieve leaf node data of the acceleration data structure from thememory, the retrieved data being associated with a geometric object;determine, based on the retrieved data, whether the ray intersects eachof plural bounding volumes enclosing different portions of the geometricobject; and performing a single intersection test between the ray andthe geometric object upon the ray being determined to intersect morethan one of the plural bounding volumes.
 2. The ray tracing device ofclaim 1 wherein the plural bounding volumes comprise axis alignedbounding volumes.
 3. The ray tracing device of claim 1 wherein adetermination that the ray intersects one of the plural bounding volumesis used as an indication that the ray intersects another of the pluralbounding volumes.
 4. The ray tracing device of claim 1 wherein theacceleration data structure comprises a bounding volume hierarchy, andthe plural bounding volumes are represented by a compressed treelet. 5.The ray tracing device of claim 4 wherein the compressed treelet definesthe plural bounding volumes for a child node, wherein the pluralbounding volumes include at least one extension bounding volume.
 6. Theray tracing device of claim 1 wherein the plural bounding volumes aredefined in world space for interior nodes at a top level accelerationstructure.
 7. The ray tracing device of claim 1 wherein a compressedtreelet portion specifies the number of said plural bounding volumes. 8.The ray tracing device of claim 1 wherein the plural bounding volumeseach bound both first and second geometric objects, and the circuitry isstructured to identify a first set of plural bounding boxes bounding thefirst geometric object and a second set of plural bounding boxesbounding the second geometric object.
 9. The ray tracing device of claim1, wherein the circuitry is a part of a server or a data center employedin generating an image, and the image is streamed to a user device. 10.The ray tracing device of claim 1, wherein the circuitry is employed ingenerating an image, and the image is used for training, testing, orcertifying a neural network employed in a machine, robot, or autonomousvehicle.
 11. A non-transitory memory storing a compressed treelet formatcomprising: a header including a field specifying a number ofaxis-aligned bounding volumes that bound a single leaf node primitivepolygon; an identification of an axis-aligned bounding volume bounding afirst portion of the single primitive polygon; and an identification ofa further axis-aligned bounding volume bounding a further portion of theprimitive polygon, the further portion of the primitive polygon notbeing encompassed by the first portion.
 12. The non-transitory memory ofclaim 11 wherein the compressed treelet is sized to fit within a cacheline, and the header includes additional fields that include at leastsome of mode, multi-box count, and plural pointers to a bounding volumehierarchy.
 13. A method of testing whether a ray intersects a leaf nodegeometric object instance that is not capable of being closely fittedwith a single axis-aligned bounding volume, comprising: (a) determiningplural axis-aligned bounding volumes that collectively more closely fitthe geometric object instance than the first-mentioned singleaxis-aligned bounding volume; (b) representing the plural axis-alignedbounding volumes in a data format; (c) using the data format to traversea bounding volume hierarchy; and (d) based on the traversal, determineintersection between a ray and each of the plural axis-aligned boundingvolumes, wherein the ray is found to potentially intersect the geometricobject instance for ray-polygon instance testing when the ray is foundto intersect any of the plural axis-aligned bounding volumes.
 14. Themethod of claim 13 wherein (c) and (d) are performed by tree traversalhardware.
 15. The method of claim 13 further comprising (e) testing theray against the geometric object instance in response to finding the rayintersects any of the plural axis-aligned bounding volumes.
 16. Themethod of claim 13 wherein at least steps (c) and (d) are performed bytree traversal hardware.
 17. The method of claim 13 wherein the dataformat specifies a compressed treelet comprising extension boundingvolumes.
 18. The method of claim 13, wherein (a), (b), (c) and (d) areperformed on a server or in a data center to generate an image, and theimage is streamed to a user device.
 19. The method of claim 13, wherein(a), (b), (c) and (d) are performed to generate an image used fortraining, testing, or certifying a neural network employed in a machine,robot, or autonomous vehicle.
 20. A ray tracing method comprising: (a)determining multi-boxes bounding different portions of a geometricobject within an object coordinate space; (b) hoisting the determinedmulti-boxes to world coordinate space; and (c) performing top levelacceleration structure intersection testing between a ray and each ofthe multi-boxes hoisted to world coordinate space.
 21. The ray tracingmethod of claim 20 wherein hoisting comprises transforming each of themulti-boxes from object coordinate space to world coordinate space. 22.The ray tracing method of claim 20 wherein (a) is performed at a leafnode and (b) is performed at an interior node.
 23. (canceled)
 24. Theray tracing method of claim 20 wherein the hoisted multi-boxes havedifferent sizes than the determined multi-boxes.